Failure during termination should always leave state as Error(Deleting)

Delete is a non-reversible operation for the user, and once the
user has indicated that they want to delete the instance from that
point on is the system's problem to complete the delete as soon as
possible.

If anything fails during the delete that the system cannot recover
from then the instance should be left in an Error(Deleting) state.
Anything else, in particular reverting to an Active(None) state, makes
it look like the system has ignored the request.

Currently InstanceTerminationFailure is explicitly caught, but the
exception does not propagate so the instance_fault wrapper does not
get a chance to log the failure.  Also terminate_instance is wrapped
by reverts_task_state which resets the state to Active(None)

This change removes the revert_task_state wrapper and catches all
exceptions so that unhandled exceptions always leave the instance
in Error(Deleting)

Change-Id: I5fb1bbd56035792f566a6e076edfe7a69df006ef
Closes-Bug: 1254122
Use utils method when getting instance metadata and system metadata

Change-Id: Idab1ba25a8ec4c7dcc9efbb9d86e7248ae3c351e
Hyper-V:Preserve config drive image after the instance is resized

If we configure config_drive_cdrom=True in nova.conf,
the config drive image will be lost after the instance
is resized on hyperV. This is because nova compute will
not copy the disk file if the file's ResourceSubType
is not 'Virtual Hard Disk'.

This commit adds 'Virtual CD/DVD Disk' into disk_resources
to preserve the config drive image after the instance is
resized.

Closes-Bug: #1250324

Change-Id: Ia125a8b132d194ddc2230754fd1b813698b410a8
Allow deleting instances while uuid lock is held

Avoid issues where locked up operations on the instance prevent it
from being deleted by the user. Reasons for the lock up are a separate
issue and will be handled elsewhere, but terminating the instance
should not affect any tasks anyway. Any modification should already
gracefully handle the instance going away.

For more discussion about the issue see:
http://lists.openstack.org/pipermail/openstack-dev/2013-October/017454.html

Partial-Bug: #1248563

Change-Id: Ife26ffefccea11d6e9ca5fbd970192b322fc04fd
Ensure that headers are utf8, not unicode

Although the api.openstack.wsgi.ResponseObject converted the headers to
utf8 strings to comply with RFC for HTTP this was not enough, since some
responses that contained unicode strings were returned, causing an error
when running nova behind an http server like Apache. This change ensures
that headers are always an utf8 encoded string.

Closes-Bug: 1254017

Change-Id: I980326f1f15907bf57060acdd132011254289079
Allow run_image_cache_manager_pass to hit db slave

Deployments that have a slave_connection configured can offload
read queries there. For deployments with no slave behavior
remains the same.

Change-Id: Ic443913883d80bdbc95385f8d3e21c0d855bb7db
Partially-implements: blueprint periodic-tasks-to-db-slave
libvirt: move unnecesary comment

Remove a commented out line of code that was not used.

Change-Id: Ie4a331c8b0d1352be82a48d6a666aafb74f7dc6b
XenAPI: Tidy calls to get_all_ref_and_rec

get_all_records is more efficient than getting them individually in the
old get_all_ref_and_rec implementation.

Replaced some cases of get_all_records with a faster and more
specific API call.

Change-Id: I4725c3f1fdebdf1737941cb7db69b19afb22df9f
XenAPI: Speedup get_vhd_parent_uuid

In many cases the VDI record has already been retrieved.
Use it rather than expensively getting a new VDI record from XAPI.

Change-Id: I469e2a415c1f1924010f47d3927ca18c45a7b8f8
Define "supported_instances" for fake compute

Allow fake computes to deploy images with hypervisor_type
defined and set to "fake" and by extension to be used in
multi hypervisor_type deployments.
Mixing fake and "real" computes on the same OpenStack
deployment allows to create many fake instances and some
working instances for testing applications on top of
OpenStack (ie: fake instances for stress tests, working
ones for functional tests).

Closes-bug: #1260771
Change-Id: Idc337c4a7ed024f236ca2b60d91e2c30f7d54536
libvirt: Review of the code to use module units

In some places we do not use the module units to refer
bytes type.

Change-Id: Ia89684bb6531bfdcee565cb1bdd33ed943ec3971
Remove copyright from empty files in nova

Sync I9043bf2f646de880afccbb64e47b257aba3ca599 to nova

According to policy change in HACKING:
http://docs.openstack.org/developer/hacking/#openstack-licensing
empty files should no longer contain copyright notices.

Change-Id: Id194e118dfdd140fda4fd00d2398d7818d590257
Partial-Bug: #1262424
Use (# of CPUs) workers by default

Update the default value of the worker configuration options to be
equal to the number of CPUs on the system.  This is the most common
value that should be used to get ideal performance from nova-conductor
or nova-api.

The following options are updated in this change:

    [default]
    ec2_workers=
    osapi_compute_workers=
    metadata_workers=

    [conductor]
    workers=

DocImpact
UpgradeImpact

Change-Id: Id8009b5fd37cef03ca66cb5bf44211039ac895ba
Use block_device_info at post_live_migration_at_destination

post_live_migration_at_destination make libvirt.xml to destination
host. when execute (live) block migration for vm made by bootable
volume, method post_live_migration_at_destination write a
libvirt.xml in destination host.
but it missed block disk information so moved libvirt.xml always
has a wrong disk information.
I add block_device_info in blockinfo.get_disk_info at
post_live_migration_at_destination.
This change will make libvirt.xml with valid block disk information.

Change-Id: I31b3f2ef5f03f9ef61524bdb7ae089b18e4cd254
Closes-Bug: 1271780
Compact pre-Icehouse database migrations <= 190.

Compacts the pre-Icehouse database migrations <= 190 into a
single migration (190_havana.py).

Pre-Icehouse users will now need to upgrade to Havana before
running any Icehouse migrations.

Tested on MySQL and PostgreSQL schemas on Fedora 19 to ensure the
new compacted migration generates an identical schema to the
previous migrate scripts in Havana.

Specific changes include:

 1) Drop migrations 181 - 184 which synced up shadow table changes.
   We now accomodate these changes in the base migration.

 2) Rename unique contraints per 185_rename_unique_constraints.py

 3) Add a new BDM table format.

 4) Add instance groups tables.

 5) add reason column to service

 6) add cells unique constraint

 7) add security groups unique constraint

Change-Id: Ibf2289f37c2376f3bd564728db5400859523c619
Implements: blueprint compact-havana-db-migrations
Compact pre-Icehouse database migrations <= 200.

Compacts the pre-Icehouse database migrations <= 200 into a
single migration (200_havana.py).

Pre-Icehouse users will now need to upgrade to Havana before
running any Icehouse migrations.

Tested on MySQL and PostgreSQL schemas on Fedora 19 to ensure the
new compacted migration generates an identical schema to the
previous migrate scripts in Havana.

Specific changes include:

  1) Add unique constraint on quotas.

  2) cleanup virtual_interfaces unique constraints

  3) Remove a bunch of volume tables (cinder cleanup)

  4) Indexes cleanup per 194_fix_152_migration_indexes.py

  5) Add unique constraint on fixed_ips.address.

  6) Add unique constraint on services columns.

  7) Add unique constraint on agent_builds columns.

  8) Add unique constraint on console_pools columns.

  9) Add unique constraint on aggregate_hosts columns.

  10) Add transport_url column to cells. (removes RPC columns)

Implements: blueprint compact-havana-db-migrations

Change-Id: Ie14e9e5de6d7d43ebabfc4680ed6434d08ecee49
Compact pre-Icehouse database migrations <= 210.

Compacts the pre-Icehouse database migrations <= 210 into a
single migration (210_havana.py).

Pre-Icehouse users will now need to upgrade to Havana before
running any Icehouse migrations.

Tested on MySQL and PostgreSQL schemas on Fedora 19 to ensure the
new compacted migration generates an identical schema to the
previous migrate scripts in Havana.

Specific changes include:

 1) Add more indexes for SQLite DB (per 201).

 2) Add unique constraint on instance_type_extra_specs.

 3) Add project_user_quotas, indexes, and constraints.

 4) Add indexes to reservations.

 5) Add locked by to instances.

 6) Add cleaned column to instances.

 7) Add host_ip/supported_instances to compute_nodes.

 8) Add missing Fkeys.

 9) Fix project_user_quotas_user_id_deleted index.

Implements: blueprint compact-havana-db-migrations

Change-Id: I63e00cd29eb4c26717b5ebd7a1ff11a01c0085de
Finish compacting pre-Icehouse database migrations

Compacts the pre-Icehouse database migrations into a
single migration (216_havana.py).

Pre-Icehouse users will now need to upgrade to Havana before
running any Icehouse migrations.

Tested on MySQL and PostgreSQL schemas on Fedora 19 to ensure the
new compacted migration generates an identical schema to the
previous migrate scripts in Havana.

Specific changes include:

  1) Add aggregate metadata unique constraint.

  2) Fix migrations index (add deleted column).

  3) Add pci_devices table.

  4) fix migrations_instance_uuid_and_status_idx index.

Implements: blueprint compact-havana-db-migrations

Change-Id: Id96e13b0ed65837761bd173c9e32ebcc1ff5f376
Make service workers gracefully handle service creation race

Right now, service workers can race to create their service record
in the database. This makes a NotFound...Exists pattern not fail
when the service record shows up due to a successful sibling.

Change-Id: I3e912beeb953c85189f44171d958e9402abc01ae
Add Network object

This adds a Network object to provide services and modeling
for nova-network and its use of the current SQLAlchemy Network
model.

Related to blueprint nova-network-objects

Change-Id: I7f943cda32fc98b3b03cdbb5d68e0c741053afaa