Support networks without gateway

Launching an instance when using network injection on a network
containing a subnet that has no gateway fails in scheduling with
a KeyError when using a legacy network_info provider and generates
an invalid network file for the others.

The following changes were done:

* The legacy transformer for network_info now always provides
  a gateway/gateway_v6 field, with a value of None if no gateway
  is available (this is similar to the other fields that are empty).
* The interfaces template was modified to include the gateway line
  only if a gateway is provided for the interface.

Fixes: bug #1207878
Change-Id: I887757a6bcba528059ace8831a7d23a31b08c630
fix libguestfs mount order when inspecting

variable `mounts` sorted value is
[
    ('/boot', '/dev/vda1'),
    ('/', '/dev/vda2'),
    ('/var', '/dev/vda5'),
    ('/home', '/dev/vda6'),
    ('/home/q', '/dev/vda7')
]
but `/boot` directory is in `/` partition.

Fixes: Bug #1210371
Change-Id: I3d9cd59237bc4fc277415104c9170d2e372a1c25
Make InvalidInstanceIDMalformed an EC2 exception

Rename InvalidInstanceIDMalformed to InvalidInstanceIDMalformedEC2
and add ec2_code attribute to make it proper EC2 API exception.

Implements: blueprint ec2-error-codes
Change-Id: I8841f79890e087b850801bf71da0512bd329d5df
Removed the duplicated _host_state = None in libvirt driver

Happened to see there was a duplicated value assignment of host state
to None in __init__() in libvirt driver. Removed one of them.

Change-Id: I4d939f301c86b369b8f740bb18c9f3f6c77a674b
Signed-off-by: Shane Wang <shane.wang@intel.com>
Clear out service disabled reason on enable for V3 API

When a service gets re-enabled, clear out the 'disabled_reason' column.
Corresponding fix for the V3 API which was commited for the V2 api
in commit 16b4efdb0d488478c6f03218931bac4c42a278e3

Fixes bug 1212217

Change-Id: I3a5cc716f6c4929abaddf457b06f70f561789f38
VMware: Ensure Neutron networking works with VMware drivers

Fixes bug 1202042

A new utility method, is_neutron, is added. This returns True
if Neutron/Quantum is used by the network_api_class.

Change-Id: Id1d1f29a02e906b98853670ea2ad589f04eba595
Fix migration 210 tests for PostgreSQL

We check index in table in _check_210() and _post_downgrade_210()
methods in a such way:

self.assertIn(('project_user_quotas_user_id_deleted_idx',
               ['user_id', 'deleted']), index_data)

But we have no possibility to get the order of columns due to SQLA
issue in reflection of indexes for PostgreSQL.
See http://www.sqlalchemy.org/trac/ticket/2767
So columns order in `index_data` variable can be wrong and test
fails in such case.

To fix this we should store colums as sets (instead of lists) if
we run tests on PostgreSQL.

Fixes bug 1213921

Change-Id: I9e10b325940becfdcff6b13fcaa4146731126a60
Make API part of instance boot use new BDM format

This patch makes the API part of the instance boot process rely on the
new BDM format. It adds additional validations that are enabled due to
using the new format.

Since the new format will allow to specify devices that will be created
as images on the hypervisor (if the hypervisor enables it), in order
to prevent a DOS, a new config option 'max_local_block_devices' was
added. This option allows the operator to set a limit on the number of
destination_type='local' block devices a user can specify per VM.

This patch also refactors and simplifies how block devices are handled
during boot, and makes sure that validation errors are caught and
transformed to expected HTTP errors.

Change-Id: I62e0807fcff0a284ff72555d107cac1c9864e46d
blueprint: improve-block-device-handling
Change RPC to use new BDM format for instance boot

This patch makes rpc calls that are the result of booting an instance
propagate the new block device mapping format.

The way this is done is by adding a 'legacy_bdm' flag to the conductor
(task) API. Since both the scheduler and the compute service propagate
the block device mapping as part of the request_spec field, a flag named
legacy_bdm_in_spec to indicate that was introduced in their respective
APIs.

Currently block_device_mapping is not used by any of the in-tree filters
in the scheduler, so it is assumed that scheduler's boot functionality
has now transitioned to the new format.

This patch also bumps RPC versions of the task, scheduler and compute
manager classes.

Finally this patch propagates the legacy_bdm_in_spec through the
run_instance callback of the compute manager so that, in case of error -
the instance is re-scheduled with the proper flag set.

blueprint: improve-block-device-handling

Change-Id: I5f25ddd4d586dda91061f065c1796be726b0ede3
Refactor how BDMs are handled when booting

This patch refactors how block devices are handled when booting VMs.
Also it introduces new block device mapping format in the run_instance
manager method of the compute service.

The refactoring is done by creating a simple hierarchy of
DriverBlockDevice classes that are meant to:

1) Provide a common place to keep information about the data that is
kept for each block device type

2) Encapsulate the logic of "attaching" the device from the compute
service point of view (driver will still need to do the actual work).

Currently we have special subclasses for ephemeral, swap, volume and
snapshot devices.

These classes also make it possible to easily provide
the drivers that will require old block device mapping format with it by
overriding the legacy() method.

Another advantage that this refactoring brings is that adding additional
block device types, like Image block devices (coming soon in a subsequent
patch) that can have destination type of either a hypervisor image or a
volume will be just a matter of extending the base class and redefining
attach to do the work of creating the volume if needed.

blueprint: improve-block-device-handling

Change-Id: I5b9c3e2d959c602fa22f49db681da918ae0adcea
Adding support for iSER transport protocol

Implements: blueprint add-iser-support-to-nova
Change-Id: I3f88b376c6d5b2682d421d3093e1f8decd263f5e
Merge "fix libguestfs mount order when inspecting"
Enforce flavor access during instance boot

The code in the servers API did not pass the context when retrieving
flavor details.  That means it would use an admin context instead,
bypassing all flavor access control checks.

This patch includes the fix, and the corresponding unit test, for both
the v2 and v3 APIs.

Closes-bug: #1212179

Change-Id: I681ae9965e19767df22fa74c3315e4e03a459d3b
Merge "Make API part of instance boot use new BDM format"
Merge "Change RPC to use new BDM format for instance boot"
Search filters for get_all_system_metadata should use lists

The search filter should have the form:
[{'foo'}: ['bar'], {'baz'}: ['quux']]

The underlying code accidentally and unfortunately allows filters that look like
[{'foo'}: 'bar', {'baz'}: 'quux'], but that is not guaranteed to work in the future.

Change-Id: Ic28612c8c843194e2f4f6918166177813fd5dee3
Merge "Support networks without gateway"
Merge "VMware: Ensure Neutron networking works with VMware drivers"
Merge "Enforce flavor access during instance boot"
Merge "Adding support for iSER transport protocol"