Support networks without gateway

Launching an instance when using network injection on a network
containing a subnet that has no gateway fails in scheduling with
a KeyError when using a legacy network_info provider and generates
an invalid network file for the others.

The following changes were done:

* The legacy transformer for network_info now always provides
  a gateway/gateway_v6 field, with a value of None if no gateway
  is available (this is similar to the other fields that are empty).
* The interfaces template was modified to include the gateway line
  only if a gateway is provided for the interface.

Fixes: bug #1207878
Change-Id: I887757a6bcba528059ace8831a7d23a31b08c630
Make InvalidInstanceIDMalformed an EC2 exception

Rename InvalidInstanceIDMalformed to InvalidInstanceIDMalformedEC2
and add ec2_code attribute to make it proper EC2 API exception.

Implements: blueprint ec2-error-codes
Change-Id: I8841f79890e087b850801bf71da0512bd329d5df
Removed the duplicated _host_state = None in libvirt driver

Happened to see there was a duplicated value assignment of host state
to None in __init__() in libvirt driver. Removed one of them.

Change-Id: I4d939f301c86b369b8f740bb18c9f3f6c77a674b
Signed-off-by: Shane Wang <shane.wang@intel.com>
Clear out service disabled reason on enable for V3 API

When a service gets re-enabled, clear out the 'disabled_reason' column.
Corresponding fix for the V3 API which was commited for the V2 api
in commit 16b4efdb0d488478c6f03218931bac4c42a278e3

Fixes bug 1212217

Change-Id: I3a5cc716f6c4929abaddf457b06f70f561789f38
VMware: Ensure Neutron networking works with VMware drivers

Fixes bug 1202042

A new utility method, is_neutron, is added. This returns True
if Neutron/Quantum is used by the network_api_class.

Change-Id: Id1d1f29a02e906b98853670ea2ad589f04eba595
Fix migration 210 tests for PostgreSQL

We check index in table in _check_210() and _post_downgrade_210()
methods in a such way:

self.assertIn(('project_user_quotas_user_id_deleted_idx',
               ['user_id', 'deleted']), index_data)

But we have no possibility to get the order of columns due to SQLA
issue in reflection of indexes for PostgreSQL.
See http://www.sqlalchemy.org/trac/ticket/2767
So columns order in `index_data` variable can be wrong and test
fails in such case.

To fix this we should store colums as sets (instead of lists) if
we run tests on PostgreSQL.

Fixes bug 1213921

Change-Id: I9e10b325940becfdcff6b13fcaa4146731126a60
Refactor how BDMs are handled when booting

This patch refactors how block devices are handled when booting VMs.
Also it introduces new block device mapping format in the run_instance
manager method of the compute service.

The refactoring is done by creating a simple hierarchy of
DriverBlockDevice classes that are meant to:

1) Provide a common place to keep information about the data that is
kept for each block device type

2) Encapsulate the logic of "attaching" the device from the compute
service point of view (driver will still need to do the actual work).

Currently we have special subclasses for ephemeral, swap, volume and
snapshot devices.

These classes also make it possible to easily provide
the drivers that will require old block device mapping format with it by
overriding the legacy() method.

Another advantage that this refactoring brings is that adding additional
block device types, like Image block devices (coming soon in a subsequent
patch) that can have destination type of either a hypervisor image or a
volume will be just a matter of extending the base class and redefining
attach to do the work of creating the volume if needed.

blueprint: improve-block-device-handling

Change-Id: I5b9c3e2d959c602fa22f49db681da918ae0adcea
Virt driver flag for different BDM formats

This patch introduces a new virt driver property
'need_legacy_block_device_info' that will tell the compute manager if
driver needs to be passed legacy block_device_info.

This patch also refactors the compute manager methods that would pass
the block device info to driver methods - to do so according to the
property of the driver. This is done by moving all the code into a single
private method _get_instance_volume_block_device_info. This method existed
before but now uses the new DriverBlockDevice classes for it's
functionality.

As it is sometimes necessary to refresh the connection info for volume
based DriverBlockDevice objects (a private manager method
_refresh_block_device_connection_info was used for this), this patch
also factors this code to a method on the DriverVolumeBlockDeviceClass,
and is now completely used like that when needed, and removes the said
method.

Finally - this patch removes a _get_block_device_info method from the
compute API class. This remained from early pre-conductor attempts to
remove DB access from the compute service, and is irrelevant now that
conductor is used. This is needed so that the source  of block device
data is always conductor and the version of data is always
known.

After this patch - it will be possible to implement changes in drivers
that will allow them to utilise the new block device data format.

blueprint: improve-block-device-handling

Change-Id: If961329061e5413824d891eada645dd8dd9cb6dd
Boot from image destination - volume

Added class DriverImageBlockDevice that implements
creating volume from image on booting VM
from image if destination_type is volume.

Adding converting to legacy if source is image and
destination is a volume.

Change-Id: I8b8a7a2e40c083c18c4808b6b340360138086c1f
blueprint: improve-block-device-handling
Adding support for iSER transport protocol

Implements: blueprint add-iser-support-to-nova
Change-Id: I3f88b376c6d5b2682d421d3093e1f8decd263f5e
Enforce flavor access during instance boot

The code in the servers API did not pass the context when retrieving
flavor details.  That means it would use an admin context instead,
bypassing all flavor access control checks.

This patch includes the fix, and the corresponding unit test, for both
the v2 and v3 APIs.

Closes-bug: #1212179

Change-Id: I681ae9965e19767df22fa74c3315e4e03a459d3b
Search filters for get_all_system_metadata should use lists

The search filter should have the form:
[{'foo'}: ['bar'], {'baz'}: ['quux']]

The underlying code accidentally and unfortunately allows filters that look like
[{'foo'}: 'bar', {'baz'}: 'quux'], but that is not guaranteed to work in the future.

Change-Id: Ic28612c8c843194e2f4f6918166177813fd5dee3
Merge "Support networks without gateway"
Merge "VMware: Ensure Neutron networking works with VMware drivers"
Merge "Enforce flavor access during instance boot"
Merge "Adding support for iSER transport protocol"
Merge "Search filters for get_all_system_metadata should use lists"
Merge "Fix migration 210 tests for PostgreSQL"
Merge "Make InvalidInstanceIDMalformed an EC2 exception"
Merge "Refactor how BDMs are handled when booting"