all_tenants=0 should not return instances from all tenants

Fixes set up of server list filters so specifying all_tenants=0
or all_tenants=False when requesting a list of servers does not
result in instances from all tenants being returned.

Change-Id: I72575e4e2cf9019cfd5bf3b07ada53cadf2001e2
Closes-Bug: #1240347
Move get_all_tenants policy enforcement to API

Cleanup to move the enforcement of the get_all_tenants policy
enforcement to the API level. The all_tenants search option in
compute/api.py is only used to trigger policy enforcement while
the logic required by all_tenants (not specifying a project_id or
user_id filter) already resides in the servers API. The flag does
not need to be passed to the db level as it is ignored at that
level anyway.

Partially implements blueprint nova-v3-api

Change-Id: I5200c8912596ffbd32c5f7c6d74939782ff70260
Enforce compute:update policy in V3 API

Now that compute/api.py:update() is no longer being called for instance
updates the policy check is no longer enforced.  This adds policy
checking back in at the V3 API level so that the compute:update policy will
continue to function as it did before.

This was fixed in I365ad8ac2845d914deefb841c314a50f4869dba0 for the V2
API.

Change-Id: I9caa3e10fe029569c6fd467e7b86103dd6590f9b
Closes-Bug: 1242955
Sync log from oslo

5fef5a8105 allow keeping of existing loggers with fileConfig
fb8075a19d Add amqp=WARN,qpid=WARN to default_log_levels
4654a5d247 Don't override default value for eventlet.wsgi.server logging
41ddd7a975 _get_log_file_path explictly return, when logfile/logdire unset
e12e6b45f8 Make openstack.common.log Python 3 compatible
572cfecb4d Make Messages unicode before hitting logging
3243cdfbe9 python3: handle module moves in log

DocImpact

Change-Id: I7bd940a9a2dd4af3d876f2acc650f18675ca177b
Clean BDM when snapshoting volume-backed instances

When snapshoting a volume-backed instance, its block device mapping is
copied entirely into the image properties of the resulting snapshot.

This may lead to some errors when booting from that snapshot because the
fields 'id', 'insance_uuid' and 'connection_info' must be generated for
each instance instead of reusing the ones from the original instance.
For example, sharing the connection_info will cause the new instance to
try to connect to the original volume instead of creating a new one.

This patch cleans all the database-specific plus the connection_info
fields from the block device mapping when creating the snapshot.

Change-Id: I4190e9519d641c61b00b5d86c3885f2ae4fa86f3
Partial-Bug: #1246327
Process image BDM earlier to avoid duplicates

If the block device mapping from the image properties are not handled at
the same time as the ones defined from the API, we might make the false
assumption that the instance has no root and failing or, in the case
we're booting from an image that only defines a BDM and has no disk,
creating a local disk BDM as root and ending up with two root devices
which is forbidden.

This patch moves the handling of the image block device mappings to the
same method were we check the ones provided throught the API. This
allows nova to decide wether the instance needs a local disk from an
image if no root device is defined in any of the block device mappings.

Change-Id: Ide95357895ab4dd1338ab5ee3ec25294af1d010b
Closes-Bug: #1246327
Refactor UnexpectedTaskStateError for handling of deleting instances

To avoid having to repeatedly unfold UnexpectedTaskStateErrors and
do seperate handling of the 'deleting' task state from other states,
everywhere this exception is handled, introduce a separate exception
UnexpectedDeletingTaskStateError that derives from the above.

This way, most handling of InstanceNotFound to handle instances gone
during some action can be made to also include this new exception.

Change-Id: Ia833c2b2361f32c7e33c84ec14d3af6231119a99
Related-Bug: #1223843
libvirt: pass instance to a log() call in the standard way

If we pass the instance in the correct way, then we get nicely
formatted logs which are easier for admins to use.

Change-Id: I27ab372a3388e2aa02491da07c616807bcbb14ef
Shrink the exception handling range

On evacuate action API, the exception handling for getting API
parameters is wide. This patch shrinks it to necessary range for
code cleanup.
In addition, this patch adds some negative tests for the lack of
"onSharedStorage" parameter.

Change-Id: Iccfa9d455758af432b5bcbb33842034f07e8fdc3
Cache compute node info in Hypervisor api

When extend hypervisor api, it would be much better to cache the
compute node info like is done for instance and flavor requests.

Partially implement blueprint pci-api-support

Change-Id: I73a96db7beb4cc0f017008f81e9f671382ad9105
Add v3 api samples for availability_zone

This patch adds api sample tests and api samples for availability_zone.
Because the detail's response use host name as the key of dict and samples
tests didn't support replacing key with variable, so startup nova's service
with specific hostname instead of uuid.

Partially implements blueprint v3-api-unittests
Partially implements blueprint v3-api-specification

Change-Id: I9090a73a8293f26d63744a10d7ffbfc887e37e7c
Add v3 api samples for scheduler-hints

Adds api sample tests and api samples for scheduler-hints

Partially implements blueprint v3-api-unittests
Partially implements blueprint v3-api-specification

Change-Id: If7bc47dc92c1e7f33f80ea567688d067c42fa219
Add V3 api samples for access_ips

Adds api sample tests and api samples for access_ips

Change-Id: I9d9249a48e7f7ccdff0146fbe5b25463d66a93a7
Conditionalise automatic enabling of disabled host

The change Ib8d67838ceb73c5b1cdc9498c17b335e9e5bb6f3 introduced
automatic behaviour to the nova-compute hosts, which will now
automatically disable themselves if they lose contact with libvirtd,
and re-endable themselves on a successful reconnection.

However, this tramples over any operational service disablement
that may be in place (on an otherwise working host) for any number
of good reasons.

As suggested in bug #1250049, this is a quick fix which attaches
a prefix to any host disable_reason string, and only re-enables
a host if that string is still present.

Fixes-bug: #1250049
Change-Id: Ia7544a357d1385db5fa92e500cb691f49436cfc9
Rename InstanceType exceptions to Flavor

Goes through and renames the various exceptions with InstanceType in the
name or message string to use Flavor instead.

Part of bp flavor-instance-type-dedup

Change-Id: I59d77086af0747f95ad61a569e784b766c8909a9
Move some libvirt specific flags into a group

This is backwards compatible via deprecated groups. This change
moves the flags defined in nova/virt/libvirt/driver.py into a
"libvirt" configuration group.

There are other flags which need to be moved, but this change
is quite large, so I will handle those in follow up changes.

Now that the flags in nova/virt/libvirt/driver.py are in a libvirt
group, we can drop the libvirt_ from in front of many of the flag
names as well.

Progresses blueprint libvirt-opts-group.

DocImpact: the following flags are moved into a "libvirt" flag
group: rescue_image_id, rescue_kernel_id, rescue_ramdisk_id,
libvirt_type, libvirt_uri, libvirt_inject_password,
libvirt_inject_key, libvirt_inject_partition, use_usb_tablet,
live_migration_uri, live_migration_flag, block_migration_flag,
live_migration_bandwidth, snapshot_image_format,
libvirt_vif_driver, libvirt_volume_drivers

The following flags have had the libvirt_ dropped from the start of
their flag name now that they are in a flag group:
    libvirt_type (now virt_type)
    libvirt_uri (now conn_uri)
    libvirt_inject_password
    libvirt_inject_key
    libvirt_inject_partition
    libvirt_vif_driver
    libvirt_volume_drivers
    libvirt_disk_prefix
    libvirt_wait_soft_reboot_seconds
    libvirt_non_blocking
    libvirt_cpu_mode
    libvirt_cpu_model
    libvirt_snapshots_directory

The name for the virt_type flag was influenced by bug 1195361,
which shows confusion as to what the flag is setting (the
virtualization type libvirt uses, not the driver libvirt uses
to achieve that).

Finally, the flag libvirt.nonblocking has been renamed to
libvirt.api_thread_pool, which is more descriptive.

Change-Id: I11e79eb5fd79c459374b9bf04707a4e6d79e993f
Move base_dir_name option to somewhere more central

The VMWare hypervisor driver is using the base_dir_name flag, but
it was defined in the libvirt driver. Given I am moving flags for
libvirt into a libvirt group, this flag needs to move somewhere
more central. It has also been renamed to the more descriptive
image_cache_subdirectory_name.

Progresses blueprint libvirt-opts-group.

DocImpact: the base_dir_name flag has been renamed
image_cache_subdirectory_name, although the old name will continue
to work for the forseeable future.

Change-Id: If016456b9f571ff0ec88621bd17e463faeef95ba
Move flags in libvirt's imagecache to the libvirt group

This change continues moving libvirt specific flags into a libvirt
specific configuration group.

Progresses blueprint libvirt-opts-group.

DocImpact: the following flags move into the libvirt group:
image_info_filename_pattern, remove_unused_base_images,
remove_unused_kernels, remove_unused_resized_minimum_age_seconds,
remove_unused_original_minimum_age_seconds, checksum_base_images,
checksum_interval_seconds.

Change-Id: Iea681c7db7efc9da0525ef8e01b4900ca71f10cf
xenapi: kernel and ramdisk missing after live-migrate

After a live migration is performed with xenserver, the kernel
and ramdisk images are not moved to destination.  This means:
* need to manually add them to the destination host
* need to manually clean them on source host

Adding the images is fairly straightforward by calling
create_kernel_and_ramdisk at post_live_migration_at_destination.
Removing images requires us to cache image ids as they are
taken from VM record which is deleted by the time the post
operation comes along.  We use the migrate_data to cache the
image data for removal.

The unit tests are adjusted to handle new calls for existing
tests.

Fixes bug 1224561

Change-Id: I741398db9ddb6764408878a11cf4c80544bbc901
Fix cells instance_create extra kwarg

When the API cell receives a message to update an instance, it'll try to
create it if it doesn't already exist in order to sync up.  However,
it's calling db.instance_create with an extra kwarg that doesn't exist
and failing.  This patch removes the extra kwarg.

The bug was introduced in this change id:

Ife5be9922db0742c8ee4f970517396be86597cce

Change-Id: I69bcb3dda2cbb566b0d69c5eac005703654ea811
Closes-bug: 1247919