image cache clean-up to clean swap disk

Currently swap backing files are never removed from the system.
The image clean-up code should recognise at least obsolete unused files
need to be removed.

Change-Id: I09519a73c7233f2dc6f46f6cd094a3e1cc3dc395
Partial-Bug: 1253991a
Get EC2 metadata localip return controller node ip

When Get EC2 metadata localip, it will return
<instanceIP, Controller_Local_IP>, but for ec2
metadata localip, it expects to return instanceIP.
This patch fixes the issue.

Change-Id: I6c42e713265ebb3138f7a94df5ee996e6f68d02f
Closes-Bug: #1334857
Type conflict in trusted_filter.py using attestation_port default value

When trusted filter (nova/nova/scheduler/filters/trusted_filter.py)
in nova scheduler is running with default value of attestation_port:

default='8443'

method _do_request() in AttestationService class has this line:

action_url = "https://%s:%d%s/%s" % (self.host,
self.port, self.api_url, action_url)

It is easy to see that default type of attestation_port is string.
But in action_url self.port is required as integer (%d).
It leads to conflict.

This change provides more tests than is required only to cover this bug
fix. This cases are testing AttestationService _do_request() method
using different status_codes and different texts returned by mocked
request method.

Change-Id: I1ad57b5bd1986360416948fd00dec22456dc29a7
Closes-Bug: #1381468
VMware: enable backward compatibility with existing clusters

When VC 5.5 creates a VM the hardware version of the VM would be set as 10.
This breaks compatibility of VM's running on an older cluster. The patch
enables the VM hardware version to be set. By default the value will not be
set. That is, it will use the current supported version. If the admin wishes
a cluster to be comptable with a specific version then they should set the
value, for example with vmx-08 to be compatible with VM's created by 5.0
and 5.1.

In the event that the cluster has mixed host version types then the VC will know
which host to place them on if the hardware version is set.

The following link describes the versions - http://goo.gl/7lhmEm

DocImpact:
The hardware version will be passed as part of the flavor extra_specs. This will
be with a key 'vmware:hw_version'

Change-Id: I67f65feceab1eab6ec4e8f4934f1c8115ed1e812
Closes-bug: #1367151
Fixes Hyper-V agent IDE/SCSI related refactoring

Hyper-V Server 2012 R2 introduces a new feature for virtual
machines named "generation 2".

Generation 2 VMs don't support IDE devices, meaning that VHDs
and DVD Drives must be attached as SCSI, instead of IDE.

Since the Virtual Hard Disks and Virtual CD/DVD Disks can be
attached to IDE controllers or SCSI controllers (generation 2
only), some constants, variables and methods have been
improperly named, having the IDE prefix.

Partially implements: blueprint hyper-v-generation-2-vms

Change-Id: Ic281d7548dfe6b2f65290f687ea0ca694db1a4d0
Sync latest from oslo-incubator

Changes include:
1523f00 threadgroup: don't log GreenletExit
a99c5f1 Report: Support Sequences and Mappings as Data
4ca5091 Fixes nits in module policy
411ba35 Simplify is_user_context method
5d43644 cliutils restore python3 string decode
a3220c5 add list_opts to all modules with configuration options
df774ff Import PublishErrorsHandler from oslo.messaging
d0a1fde Switch fileutils to use python logging
b2d35ee Use list.pop(0) to keep the code simpler
262fc82 Correct default rule name for policy.Enforcer
046bc2a Mask secrets when output conf

Removed fixtures that are no longer in oslo-incubator as the
previous reviews in this series switched to using the equivalent
fixtures in oslo libraries.

Cleaned up openstack-common.conf to include only the modules
that are still in oslo-incubator.

Change-Id: Ifd2d5bd506aeb2a9400c5075c3ca24b3c02d1cb6
Make multiple_create unit tests share between v2 and v2.1

Now there is a lot of duplicated test code between v2 and v2.1 API
tests and it is better to share them for the code cleanup.
This pach makes multiple_create unit tests share.

Partially implements blueprint v2-on-v3-api

Change-Id: Icf646efeaa8e13c6a3ec48e16523122b5af1658d
Removing the headroom calculation from db layer

It's odd that this is happening in the DB API because it's dealing
with instance quotas but maybe I'm not doing anything with instance
quotas, maybe I'm doing things with security group or fixed IP quotas.

This patch moves the headroom calculation from db layer to the
calling side.

Change-Id: I8379471eea4ea7b114c2a235224f6fe61f8f981f
Closes-bug: #1369696
VMWare: get storage policy from flavor

This is part of the blueprint vmware-spbm-support. The storage
policy will be passed as part of the flavor extra_specs. This will
be with a key 'vmware:storage_policy'. If there is no storage policy
on the flavor then the default will be used.

Co-authored-by: Radoslav Gerganov <rgerganov@vmware.com>

Change-Id: I75151f092e9424eaf346799072c8cf18a2784c16
VMware: use storage policy in datastore selection

This is part of the blueprint vmware-spbm-support. This patch adds
storage policy as an additional filter to the datastore selection.
Datastores not matching the storage policy are filtered.

Co-authored-by: Radoslav Gerganov <rgerganov@vmware.com>
Co-authored-by: Gary Kotton <gkotton@vmware.com>

Change-Id: Id934b39614bb5d26e109f6f0f58b033173bfbe70
VMware: associate instance with storage policy

This is part of the blueprint vmware-spbm-support. This patch
associates an instance with a given storage policy, if defined.
This enables a vsphere admin to check if the instance is compliant
with the storage policy.

Co-authored-by: Radoslav Gerganov <rgerganov@vmware.com>
Co-authored-by: Gary Kotton <gkotton@vmware.com>

Change-Id: I79f8b165e2a82a7335f50f2a421b10e69c02c9f0
Changed testcase 'test_send_on_vm_change' to test vm change

This patch changed the test case
NotificationsTestCase.test_send_on_vm_change to test the
vm change instead of the task change

Change-Id: I244c5eca5b5b2a254b14126d92e020f072714ade
Closes-bug: #1397247
GET servers API sorting REST API updates

Updates the v2 and v3 /servers and /servers/detail APIs to support the
multiple sort keys and sort directions (using the 'sort_key' and
'sort_dir' parameters); these parameters can be specified multiple
times to create a list of sort keys and directions. These parameters
are passed from the API layer to the compute layer, then to the
instance layer (with updated version), then to the database layers,
and then to the common paginate_query function; the paginate_query
function already supports multiple sort keys and directions. The
function signatures in these various layers are updated with new
'sort_keys' and 'sort_dirs' parameters that represent the sort keys
and directions information as lists.

This support is enabled on the v2 API by the existence of a new
'os-server-sort-keys' API extension and is always enabled in the v3
API. The extension API sample issues 2 server creates and then ensures
that both servers are listed (name is unique) in the list reply.

DocImpact: The existing v2 and v3 servers API documentation needs to
reflect these new parameters.

The nova client will also be updated to use these parameters.

Change-Id: I02baf6c3cc7d29abab132ef1726140c57e17d9b6
Partially implements: blueprint nova-pagination
Port virtual-interfaces plugin to v2.1(v3) API

Ports virtual-interfaces extension and adapts it to the
v2.1/v3 API framework. API behaviour is identical.

- unittest code modified to share testing with both v2/v2.1
- Adds expected error decorators for API methods

Partially implements blueprint v2-on-v3-api

Change-Id: Ic25ae83ecfaaa461b6ddd044e04af844d02dd6b7
Add api validation schema for image_metadata

Add schema file to do api input validation for image_metadata api.
And some unit test case fix for the 'update' action
- create accepts 'metadata' instead of 'meta'
- update accepts 'meta' instead of 'metadata'
- update_all accepts 'metadata' instead of 'meta'

Partially implements blueprint v2-on-v3-api
Change-Id: I364860d5514d94a146b56c74ff3f3950a3b8d5b3
Fix function name mismatch in test case

The function name is mismatch with the function itself.
'test_user_only' function has the difinition of 'test_user_id_only', and
'test_user_id_only' function has the definition of 'test_user_only'.
This change fixes the function name mismatch with functionality.

Closes-bug: #1397882
Change-Id: Ib7368295a3d5705ab7a49cdb5dab586bef0f1450
Merge "VMware: enable backward compatibility with existing clusters"
Merge "VMWare: get storage policy from flavor"
Merge "VMware: use storage policy in datastore selection"
Merge "VMware: associate instance with storage policy"