Fix power manager hangs while executing ipmitool

In nova/virt/baremetal, IPMI power manager uses ipmitool with '-F' option to
specify a file containing the remote server password.  When the passowrd was
empty, ipmitool ignores that file and asks user to set password in stdin.
This makes the thread stop.

This patch changes power manager to write '\0' to the password file in the
case of empty password, to force the password on impitool.

Closes-bug: #1237802

Change-Id: I44f56129a8ce2af4dda8dea35eb9c60d9518d54b
Include name/level in unit test log messages

Use a more informative format than just the raw message string in log
messages captured by fixtures.FakeLogger during unit testing.  For
example, this changes cryptic unit test output such as

    pythonlogging:'': {{{
    214 -> 215...
    done
    215 -> 216...
    done
    }}}

into:

    pythonlogging:'': {{{
    INFO [migrate.versioning.api] 214 -> 215...
    INFO [migrate.versioning.api] done
    INFO [migrate.versioning.api] 215 -> 216...
    INFO [migrate.versioning.api] done
    }}}

which instantly makes it clearer that these numbers come from database
migrations.

Change-Id: I56b6fa03892e6d865ccb036fdf75d2bb499b6c93
Add test for class InsertFromSelect

InsertFromSelect is used to generate SQL statement like
"INSERT INTO table1 SELECt * FROM table2".

Change-Id: I0fec93463be1e9930b24862724b7a6d1c8cc52af
Ensure deleted instances' status is always DELETED

Currently when deleting failed instances that were never assigned a
host, their vm_state is left in 'error' instead of 'deleted'. This is a
problem when invoking the Index Instances REST API with a Changes-Since
query parameter since that API returns both deleted and non-deleted
instances, but currently the failed deleted instances are incorrectly
marked as status:ERROR instead of status:DELETED.

This patch modifies the server ViewBuilder to ensure the status is
correctly set.

Change-Id: I614714676bab4986564b236d74f2c0d77400e48c
Closes-bug: #1243301
Rename _get_vm_state to _get_vm_status

The method is actually used to get the server's "status", so rename it
to better reflect function.

Change-Id: Ie4b0a1c76c81f6d08250a1cfd594429ef9eaf5ae
Remove unused local variables in test case

Non-functional change. Just remove the variables that had been assigned
but were never used for clarity of the code base.

Change-Id: I4c204067cff161a975df2277d6ef9d2081863bd1
Check return code of command instead of checking stderr

Command "ip link show" return a no 0 return code when passing
a no existing ip interface so we should use the return code
instead of checking stderr because stderr can contain messages
from the system even though the command succeeded.

Change-Id: Ic43e72dab3c5b6e266240b7ff7309ca3b44f8ed5
Closes-Bug: #1248699
Enable non-ascii characters in flavor names

The current implementation provided under commit e6b42d made it so that
flavor names did not have special characters (e.g. *&^%$ etc. except for
-._ and space) however it also limited names to only ascii letters.

This patch makes it so that special characters are still not allowed,
except the ones currently allowed, but it allows the use of non-ascii
characters so that flavor names can be created in locales other than
English.

Change-Id: I056451a5e8b7a49f86e0a11d0011393dc769db34
Closes-bug: #1241625
Make baremetal_deploy_helper understand ephemeral disks

Teach baremetal_deploy_helper how to run sfdisk with ephemeral disks.
baremetal_deploy_helper calls default mkfs on the epheral disk.

Change-Id: I92d473c34e9a076f5dbcd54be538410b30b10204
Partial-Bug: #1238311
Co-Authored-By: Robert Collins <rbtcollins@hp.com>
Make baremetal support metadata for ephemeral block-device-mapping

Make baremetal driver set instance['default_ephemeral_device'] to
/dev/sda1 since we are using the same device for root(sda3) and
ephemeral(sda1)

Change-Id: Ib2076198eacb0ea61706043b42d673c35d26f5d1
Partial-Bug: #1239524
Added monitor (e.g. CPU) to monitor and collect data

This patch gives a monitor interface for users to add their own monitors to
monitor on compute nodes on the compute manager.

The compute manager loads the monitors and gets the metrics, and sends to
the scheduler for utilization based scheduling.
The patch also gives a sample to collect CPU utilization data by libvirt
driver. The patch brings a new API get_host_cpu_stats() in class
ComputeDriver, and libvirt has implemented that in the patch.
For other virt drivers, it has not been implemented yet (TODO).

This is part of the blueprint utilization-aware-scheduling and the
blueprint generic-host-state-for-scheduler.

Change-Id: I4cd4f972c5c1cf6242a77aaeb9514c8a0d4faa0d
handle empty network info in instance cache

The nw_info value [] was being incorrectly equated with None. This caused
infinite recursion when api._get_instance_nw_info returned [] and was wrapped
with @refresh_cache (e.g., _get_instance_nw_info in
nova.network.neutronv2.api.API).

Fixes bug #1251792.

Change-Id: I3034fbb20e790f2d6f22c50b74a9f9dcdf9081aa
Fix libvirt evacuate instance on shared storage fails

When evacuate instance which has config drive enabled on shared storage,
disk.config file can't be created because of permision error, chowns
disk.config if it already exists.

Change-Id: I20bafc3482ead30297ad5402a09e97141816f4d1
Closes-Bug: 1247526
Periodic task _poll_unconfirmed_resizes can now use slave db

This allows the _poll_unconfirmed_resizes periodic task to use the
slave db for reads if the deployment has a slave configured. For
slaveless deployments behavior remains unchanged.

This commit advances the MigrationList object to version 1.1

Change-Id: I10bf729e92dc599819ef74a97a834c7f2df5b162
Partially-implements: blueprint periodic-tasks-to-db-slave
Add v3 api samples for shelve

Adds api sample tests and api samples for shelve

Partially implements blueprint v3-api-unittests
Partially implements blueprint v3-api-specification

Change-Id: Ib485ab042a340c4672ad01fb35d065fd52889b07
Address infinite loop in nova compute when getting network info

Move the refresh_cache decorator to get_instance_nw_info.
In fact, _get_instance_nw_info is called by the decorator
itself, potentially causing an infinite loop in case nw_info
is not of the expected value. This is also inline with
the method's purpose, as stated by the docstring. At the
same time, ensure that delete_port_for_instance and
allocate_for_instance continue to refresh the cache by
using the decorated version.

Also, add a couple of debug traces that are friendly to
the developer.

Partial-bug: 1235435
Related-bug: 1251792

Change-Id: I06f8634ea241d05ac8fbcc290adf0cb23829f3e4
xenapi: move session into new client module

This is the start of building a set of objects on top of the XenAPI
library to make a clear separation between XenAPI client and
business logic.

This is all the first steps towards pull out the client lib into oslo
incubator, so we can share code between Nova and Cinder.

DocImpact
While there is a change to the sample config file, there is no change in
the content of the config file.

Part of blueprint xenapi-driver-refactor
Change-Id: I463923a9ee289ff4b86fcf26d8c656609685857e
Add context as parameter for resume

Now for KVM when resume an instance with block storage, nova
compute will throw exception and failed to resume the VM.

The root cause is that when resume a VM with block storage,
libvirt driver needs to call conductor via rpcapi to update
block device, but the function of resume() do not have context,
this will cause RPC api failed.

Change-Id: I712777ed1d893a2b6463d30c407b0a677e37b602
Closes-Bug: #1241337
Add context as parameter for two libvirt APIs

When finish_revert_migration is called, the caller of
finish_revert_migration already includes context as a parameter,
but finish_revert_migration did not reuse this parameter and still
re-generate the context inside finish_revert_migration, we should
add context as a new parameter for finish_revert_migration so that
the functions inside of it can reuse context when needed.

For the function _create_domain_and_network in libvirt/driver.py,
it set context as an optional parameter, but context isn't really
an optional parameter for this method. We should always pass a
context down because it might be needed somewhere inside
_create_domain_and_network().

The patch includes two parts:
1) Add context as a new parameter for finish_revert_migration
2) Set context as a required parameter for _create_domain_and_network

Change-Id: I72d131fc8dce903400e3080d51ba0fc0a1320751
Closes-Bug: #1251261
Fix performance of Server List with Neutron for Admins

https://review.openstack.org/#/c/30048/ optimized the
SecurityGroupsOutputController for the case where the server
list only contains one server, but in all other cases the current
code calls the Neutron driver in a way that makes it retrieve
all ports and security groups visible to the tenant.

For users with a Neutron admin role this retrieves all ports
and all SecGroups in the system, which on a large system is a
major performance issue and often leads to client timeouts.
Normally these users have further qualified their query to a
specific tenant or host, or are maybe just trying to get their
own list of servers.

If instead we pass the pre-filtered list of servers to Neutron
it can query for ports associated with just the device IDs of
the servers.  We then use that list of ports to build a
specific list of security groups IDs to query for.

This approach means that we can remove the initial
optimization for a single server case.

Closes-Bug: 1228384

Change-Id: I6f72533056e4f336f7578cc883fd1a125c2048a9