Fix migration 147 downgrading with no aggregates

Add some extra bulletproofing to downgrading migration 147 where
there may have been no aggregates.

Change-Id: I6e4aabf4fff462e9ad53083621077ce0e0a80a43
Fix docstring for disk_cachemodes

The docstring for the disk_cachemodes option implied a syntax that is
incorrect for a ListOpt.  Change the example to match exactly what you
would put in the config file for it to work.

Change-Id: Ie999d2799dba6c47facdc771e1cf6081d28d6b85
Closes-Bug: #1106423
(cherry picked from commit 1df9b26b51ff3a153b93d79e25bef454e08a8e38)
Fix difference between mysql & psql of flavor-show

If you create a new flavor using a deleted flavorid,
the result of flavor-show is different between mysql and psql.

This issue is caused by the different processing for index in each db.

We need to specify the sort-type to avoid the issue.

Change-Id: Ib1029e80c1b981e1ec86d954b63f83650c9b1cc1
Closes-Bug: #1288636
(cherry picked from commit 4bbb4d88595cfd5f75528d8789be94d0ec9fbe62)
Update aggregate should not allow duplicated names

It is not allow to create 2 aggregates with the same name
but if update an existing one with a name already used
by other aggregate that check should be made.
This patch doesn't allow to change the name to an existing
one.

Change-Id: Icfe315594beaec94229e5a4a4c1bb80fc366b66e
Closes-Bug: #1288296
(cherry picked from commit 2b2decfd1b5b0d68813bc7e5b5e4349a1ab592aa)
Add context as parameter for two libvirt APIs

When finish_revert_migration is called, the caller of
finish_revert_migration already includes context as a parameter,
but finish_revert_migration did not reuse this parameter and still
re-generate the context inside finish_revert_migration, we should
add context as a new parameter for finish_revert_migration so that
the functions inside of it can reuse context when needed.

For the function _create_domain_and_network in libvirt/driver.py,
it set context as an optional parameter, but context isn't really
an optional parameter for this method. We should always pass a
context down because it might be needed somewhere inside
_create_domain_and_network().

The patch includes two parts:
1) Add context as a new parameter for finish_revert_migration
2) Set context as a required parameter for _create_domain_and_network

Closes-Bug: #1251261
(cherry picked from commit 81fc3967bca0f154d72d5de7875b646e1b6c47c8)

Conflicts:
	nova/tests/compute/test_compute_mgr.py
	nova/virt/vmwareapi/vmops.py

Change-Id: I72d131fc8dce903400e3080d51ba0fc0a1320751
Make cells 'flavorid' for resizes

Cells should use 'flavorid' (not 'id') for resizes.

bug 1241222

Change-Id: I57b997425f183f25d3ca032c7d8c35c1f49e3afc
(cherry picked from commit 034df6e875d6c0b590f3cfb802574c2f982fd534)
libvirt: Custom disk_bus setting is being lost on hard_reboot

Currently, a non-default disk_bus setting of and image will not be used,
when powering on an instance (or _hard_reboot).

hard_reboot() should use image_meta, if it exist, when constructing
the disk_info

Closes-Bug: #1255449

(cherry picked from commit 0fe3001d639df81ae4e5f77bcfc32bc1bffb014e)

Conflicts:
	nova/virt/libvirt/driver.py

Change-Id: I477f69c47515ad22165421b4d23ac865c7869873
Add declaration of 'refresh_instance_security_rules' to virt driver

The method 'refresh_instance_security_rules' was not defined
in the virt driver ComputeDriver class. In cases where the
underlying virt driver does not support security groups
an exception would be encountered indication that the driver
class does not have the specific attribute.

In addition to adding the method the patch set also catches the
exception NotImplementedError and logs a warning.

Closes-bug: #1269448
(cherry picked from commit 5a98d7b61fa6adb1557e69f2f234406be0e2739c)

Conflicts:
	nova/tests/virt/hyperv/test_hypervapi.py

Change-Id: Ia36b0dfb54a3ede86467d87f0650a0902adb6d46
Always filter out multicast from reflection

Instances will try IPv6 neighbour discovery via multicast
even in an IPv4-only cloud and can throw errors
if they see inbound packets from their own MAC address

Closes-bug: #1229625

Change-Id: I3539e788fe07519d87ce7c3800c5d38b7bd99d3b
(cherry picked from commit 9c044d2c94812e18cf84927fbf719cd073fe6c4f)
Fix incorrect root partition size and compatible volume name

When using rbd backend, it needs to resize volume size if size presents. Else
we will get wrong image size.

Rbd volume always uses uuid as the main pattern of volume name, now change to
it is necessary.

This patch also include a follow up fix of the original patch.
https://review.openstack.org/#/c/92512/

Fix bug 1219658

(cherry picked from commit 1b66a47b5f4cbcdf1a9a1ec38532474588a3fee2)

Conflicts:
	nova/virt/libvirt/imagebackend.py

Change-Id: I3dc8d639b066eac35d190b8d4687c41c9f4d5824
Fixes NoneType vcpu list returned by Libvirt driver

* libvirt dom.vcpu may return a NoneType if vcpus dont exist

Change-Id: Iee10c2559ceea636bc4896595962e3470228ad35
Closes-Bug: #1274758
(cherry picked from commit 82821b407ce7a226918758876abca46b727d9706)
Avoid re-adding iptables rules for instances that have disappeared

The remove_filters_for_instance() method fails silently if the
instance's chain is gone (i.e. it's been deleted). If this
happens while we're refreshing security group rules, we will
not notice this case and re-add stale rules for an old instance,
breaking our firewall for new instances.

This adds a quick check after we've captured the lock to see if
the associated chain exists, and bails if it doesn't.

Conflicts:
	nova/virt/firewall.py

Required changes:
- unit test was updated because in Havana, IptablesFirewallDriver has
  .instances instead of .instance_infos, and a separate .network_infos
  attribute.

Change-Id: Ic75988939f82de49735d85fe99a9eecd4baf45c9
Related-bug: #1182131
(cherry picked from commit 6aa368b99249d01f8fd7183c15d11986ad6a6fb7)
(cherry picked from commit 132d5a2064cfdc579a2e51f07e10e8eafbdf6327)
Set is_public to False by default for volume backed snapshots

This update results in the same behavior that is already implemented
in _create_image where the meta_data for is_public is set to False.

Change-Id: Ie0720d5c847871527db7d9477ca6cb05abe7a1a5
Closes-Bug: #1255316
(cherry picked from commit 0530e02004578a36039111beddf472fcf3e48df6)
Block sqlalchemy migrate 0.9.2 as it breaks all of nova

This is breaking nova unit tests

Closes-Bug: #1368391
(cherry picked from commit 9c2d6f21854bc79f03095163b8133a9fec68f9f2)

Change-Id: Ic82d787e41f7b6d6153d3c6b19902cc11a1c43bb
Sync latest process and str utils from oslo

This backport the necessary changes to fix both issues:
* Make execute method clean password in exception
* Make sure mask_password works properly

Backport in oslo-incubator: 4cf1a2a158f7c3c21799bf2e6ba0e7b5fbc25d62

Closes-Bug: 1343604
Closes-Bug: 1345233
SecurityImpact

Change-Id: I3b49b1d667f6ade9ae3f6765d735440a3e838917
Merge "Set is_public to False by default for volume backed snapshots" into stable/havana
Merge "Fix migration 147 downgrading with no aggregates" into stable/havana
Make floatingip-ip-delete atomic with neutron

The infra guys were noticing an issue where they were leaking floating ip
addresses. One of the reasons this would occur for them is they called
nova floating-ip-delete which first disassocates the floating-ip in neutron
and then deletes it. Because nova-api makes two calls to neutron if the
first one succeeds and the second fails it results in the instance no longer
being associated with the floatingip. They have retry logic but they base it on
the instance and when they go to retry cleaning up the instance the floatingip
is no longer on the instance so they never delete it.

This patch fixes this issue by creating a new method
disassocate_and_delete_floating_ip which contains the different logic
needed to atomically do this for neutron rather than putting the logic
in the api layer to do something differently if one is using neutron.

This patch also adds additional code coverage for nova-network in the
delete floatingip case.

Closes-bug: 1356157

Conflicts:
	nova/api/openstack/compute/contrib/floating_ips.py
	nova/network/base_api.py

Change-Id: I53b0c9d949404288e8687b304361a74b53d69ef3
(cherry picked from commit 5f391087d1d75dd4e7532252c312256e9c3ea2b9)
Add Hyper-V driver in the "compute_driver" option description

The description of the option "compute_driver" should include
hyperv.HyperVDriver along with the other supported driver.

Conflicts:
        nova/virt/driver.py
        nova/virt/hyperv/__init__.py

Closes-bug: #1347777
(cherry picked from commit b856bc5e348cc69923e3ee815303804323730026)

Change-Id: Ic7e16fd5154609987b2ef0b6f8ee52619ccf489c
db: Add @_retry_on_deadlock to service_update()

Add the _retry_on_deadlock decorator to the service_update() method of
nova's sqlalchemy API.  This decorator is scattered throughout the
sqlalchemy API for other methods that may encounter this error.  The
referenced bug report shows a trace from where it occurred on this
method.

Change-Id: I93b370d6457d2e85493be01a62a76404d228a6fa
Closes-bug: #1370191
(cherry picked from commit 61d54f2652424279af81c4084e87651044fac4bd)