XenAPI: Refactor Fake to create pools, SRs and VIFs automatically

Required for changing the default SR to be the pool default - see
I116f65ece94f839e63b085480ba4f8974119d2bb
This also fixes some minor bugs in fake.py shown up by the tests

Change-Id: I19fc0747648b97a0912d28008981a84f70107d7b
Use valid IP addresses values in tests

Currently we use strings like 'fake_address', 'localhost', etc
as IP addresses values in DP API tests. This works great when
running tests on SQLite or MySQL backends, where IP addresses
are stored as plain strings. But in PostgreSQL there is a special
column type (INET), which checks inserted values to be valid IP
addresses, so tests fail.

This patch ensures we use valid IP addresses values instead of
arbitrary strings in DB API tests.

Blueprint: db-api-tests-on-all-backends

Change-Id: I2026bab612fcac09b245f6745810b6200f1f2443
Change EC2 client tokens to use system_metadata

An EC2 client token is created as a result of RunInstances. The initial
implementation called create_tags which uses per-instance metadata.
However, we can't update per-instance metadata while an instance is still
building because put the metadata updates in the messaging queue.

We avoid this problem by using system_metadata instead of per-instance
metadata. As an added bonus, users can't tamper with the token via the tags
API.

Fixes bug #1202404

Change-Id: I8ff28d75154a30fe3b521926453b9e9cbdd9fead
Raise exception if both port and fixed-ip are in requested networks.

When both fixed-ip and port (uuid) are specified as part of the
request during server creation, the user specified fixed-ip must
be ignored because it has already been allocated for the port. For
compute osapi v3, we raise a BadRequest Exception to ensure the user
understands why he/she is not getting what he/she asks for.

This patch also address some minor nits in other error messages.

Fixed bug #1190777

Change-Id: Iaf606637a8acf9d20fc6cae0bed5c9a3ea0bf611
Fix flavor v3 follow API v3 rules

* Change the parameter naming style for index and detail
* Add decorator expected_errors

Partially implements bp v3-api-extension-versioning

DocImpact

Change-Id: Iae6fd27e665388f4e6b24a3685ea7ed39e78a89d
Fix extension rescue follow API v3 rules

* Add decorator expected_errors
* Change 'adminPass' to 'admin_pass'

Partially implements bp v3-api-extension-versioning

DocImpact

Change-Id: Ibfc78801bf91768c62f2b055e4f663396d66f66d
xenapi: Fix console rotate script

First time run of the console rotate script failed, due to the fact that
the logs did not exist at that time.

Fixes bug 1204440

Change-Id: I49b5569c1ba3349d911caace42685fa0f19a1f66
Set VM back to its original state if cold migration failed

If migration of a VM in the SHUTOFF state failed due to
NoValidHost, the VM was erroneously set to ACTIVE.

The fix sets the VM back to its original state instead.

Fix bug 1204905

Change-Id: I06f6a2e8a05e715c505a24d3013f482eaea3df8b
Don't pass empty image to filter on live migration

If live migration for a volume backed instance without an image supplied
is initiated with no destination supplied, glance client show will not
be called and the image variable will be set to None.

However if passed to filters like that as part of the request_spec
dictionary, None value will cause one of the default filters - the
ImagePropertiesFilter to throw an exception in it's host_passes method,
as it expects an image to be a dictionary.

We solve this by leaving out the image entirely before engaging filters,
and not attempting to get it if it was not supplied to the instance.

This bug was previously fixed (however not fully, as the None image
would still cause an exception) in the following patch:
Ie683a3ca5d6430c52ead77b41f98fbcb4114ea1e, but  was accidentally dropped
in I6de33ada6dc377e20f8df07da92244f2c150b9fe.

Fixes bug 1170596 (again)
Fixes bug 1199811

Change-Id: I0239b6ac5640d075f937f9690eca4ce54f534117
Fix HTTP response for PortNotFound during boot

If a non-existent Quantum port is passed as a network parameter during create
instance, Nova API should return an HTTP BadRequest.

Fixes bug #1195468

Change-Id: I5914fda076fdd3ff20c81984364ca725edae1a78
Clarify instance_type vs flavor in nova-manage

Make it clear that we are only using the term flavor now.

Partially implements bp flavor-instance-type-dedup

Change-Id: Ic26349d70fd99a20635f3ae45041c66b300a0513
Add latest oslo DB support

The common oslo code is backward compatible.

The database configuration variables are now in a new section
[database].

DocImpact

Change-Id: I978493f692179ecf09ea9ed98d521b3e65394880
Fixes Hyper-V issues on versions prior to 2012

Fixes bug: #1203059

Hyper-V 2012 introduced live migration support and a WMI namespace that
can be used to handle iSCSI targets.

The Hyper-V driver was not properly recognizing versions of the os prior to
2012 causing an exception during startup due to the usage of unsupported
features.

Unit tests have been added to test the behaviour of the driver with
different os versions.

Change-Id: Ie2c2ba6c9aee1ea634c79ca3391c96fe42db341a
Add basic BDM format validation in the API layer

Adds the basic sanity checking for the new BDM data format when it's
passed in to the servers create method. Also make block_device routines
that raise IvalidBDMFormat error add a brief explanation to it that can
be reused to send more informative validation errors to the user.

More comprehensive validation that will take a look at the request as a
whole, and take into consideration all the metadata, will be done in
the compute API create method in the subsequent patches.

blueprint: improve-block-device-handling
Change-Id: Ie4c2c9a219be91faa07106f820a728ca9ac42e1b
Bypass queries which cause a contradiction

This addresses the cause of the log lines like:

SAWarning: The IN-predicate on "instance_system_metadata.instance_uuid"
was invoked with an empty sequence. This results in a contradiction,
which nonetheless can be expensive to evaluate. Consider alternative
strategies for improved performance.

for instance_metadata, instance_system_metadata, and instance_faults.

It especially helps the case of a call to /servers/detail when there are
no servers to list because each of those tables is scanned despite a
guarantee that there will be now rows.

Bug 1204916

Change-Id: I62e65c4409d3e3ab8373eb66450414565c222547
Fix postgresql failures related to Data type to API-v3 fixed-ip

Postgresql is kind of sensitive about different kind of data in a query, so it
was failing in cases like looking for an ip using a filter which wasn't a valid
ip or ids values greater than the id type on the table.

This patch fix this behavior to fixed ips in API-v3.
In v2 this was fixed by change I83d532c28c9aec690e8e1ffad8b58e71d619d728

Fixes bug 1182754
Fixes bug 1204503

Change-Id: I9aa14bea2f29f1df85dfd1f07eb0b1947d2c1100
Set lock_path in tests

When lock_path is not set, the synchronized decorator doesn't work
properly.  This change is required so that synchronized can throw
an exception when that happens instead of failing silently.

Change-Id: I341a522e59fda3feb8e41379d06ad3c8f97e5597
Sync lockutils from Oslo

90b6a65 Fix locking bug
27d4b41 Move synchronized body to a first-class function
15c17fb Make lock_file_prefix optional
1a2df89 Enable H302 hacking check
b41862d Use param keyword for docstrings

Fixes bug 1065531
And bug 1162047

Change-Id: Ide79292fae6f779ecd4ac166d68c8f10ca728409
Remove the monkey patching of _ into the builtins

Previous _ was monkey patched into builtins whenever
certain modules were imported. This removes that and
simply imports it when it is needed.

Change-Id: I0af2c6d8a230e94440d655d13cab9107ac20d13c
Use RetrievePropertiesEx instead of RetrieveProperties

Fixes bug 1183654

The fix also adds a configuration variable enabling the user to
configure the maximum amount of objects that can be retrieved by
the aforementioned function.

When using RetrievePropertiesEx the server may allocate resources
in case additional calls are made. These are addressed by reading the
extra resources using ContinueRetrievePropertiesEx (if necessary)
or CancelRetrievePropertiesEx (when the necessary result has been found).

DocImpact

Change-Id: I894d9698461b0ce22b01211fd196f6c32899a8fd