Fix inaccuracies in the development environment doc.

Change-Id: If2b0884de54b75e40896cee9b4fb5eb805de2624
Allow fixed to float ping with external gateway.

If you ping an a floating ip from an instance with only a fixed ip, the
traffic will be sent to the default gateway. If the default gateway has
a route to the floating ip, it will then be sent to the host of the
instance with the floating ip. The source address will be on the fixed
network, so if the floating addresses are on a different network, the
receiving host to drop the packet due to the default rp_filter.
Essentially, the route for the fixed range is on a different interface
so it the kernel assumes the packet is spoofed and drops it.

In order to fix this issue this patch adds a new config option called
force_snat_range. Traffic that originates in fixed_range and has a
destination in force_snat_range will always be routed. This forces
the packet to hit the fallback floating roule and be snatted to
routing_source_ip. This means the traffic will originate from the
proper network and the packet will make it through.

DocImpact

Fixes bug 1122335

Change-Id: I7e31c25d37a5c4fd02c8238aec58a28af3fd7f6a
Added a service heartbeat driver using Memcached.

Today the heartbeat information of Nova services/nodes
is maintained in the DB, while each service updates the
corresponding record in the Service table periodically
(by default -- every 10 seconds), specifying the timestamp
of the last update. This mechanism is highly inefficient
and does not scale. E.g., maintaining the heartbeat
information for 1,000 nodes/services would require 100 DB
updates per second (just for the heartbeat).

This patch adds nova.servicegroup.drivers.memcached, a
service heartbeat driver using Memcached. You can reduce
DB updates with it.

blueprint memcached-service-heartbeat

Change-Id: I60bdb1cfbce1fea051f276ebfd6ccc4ad8fe6d2b
libvirt: Fix LXC container creation

Recent changes to the blockdev creation introduced in
7be531fe9462f2b07d4a1abf6687f649d1dfbb89 caused a regression
when LXC containers were being created.

When the image is being fetched from glance, the disk mappings
for the disk bus, disk information, etc is checked before being
copied. However since LXC does not understand this concept it was
not passing any information, so the image was never being copied.

Send some basic information about the disk when the instance is being
created in order for the image to begin. Fixes LP: #1126348

Change-Id: I90daf853903737013be0cd23f78c7f61e4e15a77
Signed-off-by: Chuck Short <chuck.short@canonical.com>
Fix instance can not be deleted after soft reboot

The reason is that:
1. soft reboot will wait for instance become to 'shutdown', and then start it
2. delete operation also wait for this, and then clean up the instance
3. if soft reboot found the instance become to 'shutdown' firstly, it will
start it immediately
4. then the delete operation will go to the _wait_for_destroy loop, and the
loop may be endless
5. when we delete the instance again, because the lock was hold by the delete
operation before, this one will wait the lock and don't implement actually.
So the domain id is checked during _wait_for_destroy loop, if it changed and
the instance is still running, we need to destroy it again.
If the domain is booted after _wait_for_destroy, it may result in
unfilter_instance failed because the nwfilter is in use, so doing the same
thing as above.

Fixes Bug #1111213

Change-Id: I98dc902dd86fa828f5821465c611953e08f9f637
Prevent the unexpected with nova-manage network modify.

"nova-manage network modify" can misinterpret arguments if the user
provides values where they are unexpected, specifically the
--disassociate-project and --disassociate-host options. If the user
provides a project or hostname value for either of these values, it will
be interpreted as a positional argument and modify handler may behave as
if other options were provided (e.g. project or host).

This change handles the presence of a project or host option along with
--disassociate-project or --disassociate-host as an error, printing a
message and exiting immediately with an error code. While there are
other approaches (e.g. perform the disassociate modifications while
ignoring the others), preventing unexpected or inconsistent results by
doing nothing seems safest.

Change-Id: Iff7817fe3ed19a4ff330b4029f9c558d3b405e9b
Fix network list and show with quantum.

The quantum driver was returning data in a format that was not
compatible with the extension. This munges the data and handles
fields that don't exist so the extensions still work.

Fixes bug 1076519

Change-Id: I5e00bfa72e8ce1d486f9aa4dddbe1080ff74cd6a
Change all instances of the non-word "inteface" to "interface"

Change-Id: I64f92a6346461670330a0c6e066b6685d4fc1b06
Ensure rpc result is primitive types.

This new rpc method was not ensuring that the result was converted to
primitive types.  This would cause it to break when using an rpc backend
other than impl_kombu.

Change-Id: I0a21b559d12ff7d0c96341fcec887b2436753eda
Fibre channel block storage support (nova changes)

implements blueprint libvirt-fibre-channel

These changes constitute the required libvirt changes to
support attaching Fibre Channel volumes to a VM.

This requires a few new packages to be available on the system
sysfsutils -- This is needed to discover the FC HBAs
sg3-utils -- this package is needed for scsi device discovery
multipath -- This package is needed for multipath support.

Typical Fibre Channel arrays support exporting volumes via
multiple ports, so multipath support is highly desirable
for fault tolerance.  If multipath is not installed,
the new FibreChannel libvirt volume driver will still work.

If multipath is enabled, the new Fibre Channel volume driver
detects each of the attached devices for the volume, and
properly removes every one of them on detach.

In order to use this, the cinder volume driver's
initialize_connection will simply return a dictionary
with a new driver_volume_type called 'fibrechan'.

The target_wwn can be a single entry or a list of wwns
that correspond to the list of remote wwn(s) that will
export the volume.

return {'driver_volume_type': 'fibre_channel',
        'data': {'target_lun', 1,
                 'target_wwn': '1234567890123'}}

or

return {'driver_volume_type': 'fibre_channel',
        'data': {'target_lun', 1,
                 'target_wwn': ['1234567890123',
                                '0987654321321']}}

Change-Id: Ifccc56f960ef434f7cb56a9367e4cad288358440
Retry bw_usage_update() on innodb Deadlock

Adds a new decorator _retry_on_deadlock() to sqlalchemy api.  This patch
makes bw_usage_update() use it.

Fixes bug 1129622

Change-Id: I0293c62d2dd5ac036445bc639cabbd05ba016e83
Add support for instance disk IO control.

Part 3 for blueprint quota-instance-resource, by setting exposed params
"disk_read_bytes_sec, disk_read_iops_sec, disk_write_bytes_sec,
disk_write_iops_sec, disk_total_bytes_sec, disk_total_iops_sec"
through nova flavor-set, user can implement instance disk IO control
policy.
for example:
nova flavor-set m1.medium set disk_write_bytes_sec=10240000,
then the max disk write IO is 10MB/s for VM user.

partly implement blueprint quota-instance-resource
original reviews https://review.openstack.org/#/c/14151/
DocImpact

Change-Id: I8605acf61d7ddbfdaf6eee3ab135898df2449484
Add support for instance CPU consumption control.

Part 4 for blueprint quota-instance-resource, by setting cpu_shares,
cpu_quota,cpu_period through nova flavor-set with admin credentials,
 the cloud admin can implement flexable policy on VM vcpu to physical
CPU consumption ratio.

for example:
nova flavor-key m1.low_cpu set cpu_quota=10000
nova flavor-key m1.low_cpu set cpu_period=20000
the instance of m1.low_cpu can only has a max of 50% CPU consumption
of a physical CPU computing capability.

original reviews https://review.openstack.org/#/c/14151/
DocImpact
partly implement blueprint quota-instance-resource.

Change-Id: I9ea1229ee78a4d706b1a18c0f4f15157611952e4
Fix leak of loop/nbd devices in injection using localfs

Call do_teardown() rather than do_umount()

bug 1130080

Change-Id: I0dd4e4990f75917628007fd380f7df531d9a79e5
Documentation cleanups for nova devref

This patch contains a number of cleanups of the nova devref,
mostly related to outdated content.

1) remove outdated todo items from network
=> these have been long covered in the manuals

2) remove outdated multinic docs and images
=> this is now better covered in:
http://docs.openstack.org/trunk/openstack-compute/admin/content
/using-multi-nics.html

3) remove outdated cloudpipe docs, confs and scripts
=> This is now better covered in:
http://docs.openstack.org/trunk/openstack-compute/admin/content/
cloudpipe-per-project-vpns.html

4) remove outdated networking docs
=> These were marked as 'legacy' more than 2 years ago

Change-Id: I9321335031b4581c603a6f31c613e1b620d468a6
Fix key check in instance actions formatter.

The 'key in object' check doesn't work for sqlalchemy models so nothing
was getting copied over during the format.  This patch switches to using
get() to retrieve a value.

Bug 1126593

Change-Id: I726c2a624928247de41a077c23fe80742cbf9044
Add basic infrastructure for compute driver async events

This introduces the basic infrastructure to allow compute
drivers to emit asychronous events for processing by the
virt API. Initially the only event defined is a "lifecycle"
event which allows the virt API to immediately detect the
start and stop of virtual domain instances, without needing
to frequently poll the compute API.

The base compute driver gains a method enabling the manage
to register a callback to receive events, and another method
to allow driver subclasses to emit events.

Blueprint: compute-driver-events
Change-Id: Ic9abcd3f829c106e840538a01376862ab5b3485b
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Enhance IPAdresses migration tests

This enhances IPAddresses tests to check if the migration goes fine with longer
IPv6, and with ipv6 that can be shortened (postgresql uses shortened form,
mysql and sqlite doesn't).

Also enhances cidr tests to check migration of shortenables cidrs, and cidrs
that postgresql suppress its prefix, like 10.0.0.0/32 cause it is pretty much
the same as 10.0.0.0

Partially implements bp migration-testing-with-data

Change-Id: I11dfc8fda7034a46a6bc067294ac8e88ca43b508
Add support for lifecycle events in the libvirt driver

This wires up the libvirt compute driver to process async
events from libvirtd and emit lifecycle events. This makes
use of the native libvirt event loop to handle I/O processing
which requires a native thread.

The native thread uses a queue + self-pipe to forward events
onto a green thread. The green thread reads events off the
queue and uses the emit_event method to dispatch them to the
compute manager.

Blueprint: compute-driver-events
Change-Id: Icd2cb7081adde10420ae55beebe60350afe21379
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Handle lifecycle events in the compute manager

Update the compute manager to register an event handler
with the compute drivers. The initial handler will process
lifecycle event notifications, to allow prompt synchronization
of DB & hypervisor power states. This means that if a guest
admin does 'shutdown -h now' the libvirt driver in Nova will
immediately notice that and mark the guest as SHUTOFF rather
then leaving it RUNNING until the periodic task runs sometime
in the next 10 minutes

Blueprint: compute-driver-events
Change-Id: I39a6d450a6d6384b8be58aceea98e4395185cc38
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>