Fixes get_vm_storage_paths issue for Hyper-V V2 API

Add get_vm_storage_paths function to vmutilsv2.py

get_vm_storage_paths implementation on Hyper-V V2 API changed
compared to previous V1 API. In previous V1 APIs, using
Msvm_ResourceAllocationSettingData class can get storage
paths, but in V2 API, only using Msvm_StorageAllocationSettingData
class can get the storage paths.

Closes-Bug: #1248463

Change-Id: Ica73221524f162a6ffcd9dc64ee33c85fb5ad31d
Fix a lazy-load exception in security_group_update()

When security_group_api is nova, the update_security_group api raises http
500 error: group rules fails with lazy load exception. Because the rule
infos of the security group will be return in response body, but the rule
attribute hasn't been preloaded.

Closes-Bug: #1244018
Change-Id: Ie07879ac22e4add90f75968778bf334915e8349c
Check Neutron port quota during validate_networks in API

Unless ports are passed into Nova it will create ports on the
requested networks as part of the network allocation in
the compute manager. However if the user exceeds their
port quota the instance will end up in an Error state, having
first been re-scheduled a number of times.

It would be much better if the quota failure was detected as
part of the network validation in the API server, so that an
error can be reported to the user and the creation failed.

A full fix would include reserving or creating the ports at
this stage, but there is no reservation mechanism in the
Neutron API, and port creation depends in some cases
on mac addresses only available on the compute manager.

Instead this change just validates the quota and adjusts the
max_count to be consistent with that quota, which doesn't
guarantee that the create will work, but does catch the
majority of cases.

Refs bug: 1172808

Change-Id: Iaaee059a6746fad68049712f94b2f8cfea6ab8dc
LXC: Image device should be reset in mount() and teardown()

When fixing bug#1208387, image device name is returned in function
setup_container. But it's missed to reset in function mount() and
teardown().

This patch is to fix it. We now just return the device name when
mounting disk image during setting up container and remove the
member variable 'device' from Class _DiskImage added in last patch.

Closes-bug: #1243108
Related-bug: #1208387

Change-Id: I2aaed6f3367edce4fb23b78f0e979440954609a8
Setup destination disk from virt_disk_size

When running live-migration --block-migrate on a qcow2 backed
VM without cow image, the destination qcow2 file should be created
with the virtual disk size. For raw images, the virt_disk_size
is set to disk_size to ensure that virt_disk_size is always the
size of the disk that should be re-created.

Update unit tests to be more strict and check for sizes to be correct.

Closes-Bug: #1257355

Change-Id: Ie3be46024f06b9f59af92f5e3918a1958386d4f1
libvirt: consider minimal I/O size when selecting cache type

The "none" cache type is not usable when a file-system
has stricter alignment and minimal I/O size requirements,
for example the xfs file-system created with 4kiB sector size,
requires all DIRECT I/O operation has to be done with
 integer multiple of 4 kiB count/size.

qemu assumes direct I/O is doable with 512 byte minimal size,
so the  _supports_direct_io method needs to check is
the direct I/O doable with 512 byte size.

Change-Id: I5e1ba4e8afa87519cb0ed1120c213f7c318ccef2
Make it possible to override test timeout value

Currently, we are setting the test timeout value globally
in .testr.conf. It could be useful to be able to override
it per test case (e.g. for migrations tests which usually
take much longer to run than any other tests).

Related-Bug: #1216851

Change-Id: Icee09609f76f4439f6d981641be521d8d695a77e
Give migrations tests more time to run

Migrations tests usually take longer to run that most other
tests. Increase the time out value just for migrations tests,
so that we don't hit various time out errors running unit tests
when CI test nodes slow down a bit.

Closes-Bug: #1216851

Change-Id: I48d7c125397cd282d52eb8e1a1ccb699188d8f14
VMware: fix bug when more than one datacenter exists

In the case that there was more than one datacenter defined on the VC,
then spawning an instance would result in an exception. The reason for this
was that the nova compute would not set the correct datacenter for the
selected datastore.

The fix also takes care of the correct folder selection. This too was a
result of not selecting the correct folder for the data center.

The 'fake' configuration was updated to contain an additional data
center with its on datastore.

Closes-Bug: #1180044
Closes-Bug: #1214850

Co-authored-by: Shawn Harsock <hartsocks@vmware.com>

Change-Id: Ib61811fffcbc80385efc3166c9e366fdaa6432bd
XenAPI: Speedup host_ref cannot change - get it once

The host we run on is fixed - getting this each time from XAPI
is expensive

Change-Id: Ifee9a778f3ed855ef52b001de3fc2ebbd0b972cf
Compact pre-Icehouse database migrations <= 170.

Compacts the pre-Icehouse database migrations <= 170 into a
single migration (170_havana.py).

Pre-Icehouse users will now need to upgrade to Havana before
running any Icehouse migrations.

Tested on MySQL and PostgreSQL schemas on Fedora 19 to ensure the
new compacted migration generates an identical schema to the
previous migrate scripts in Havana.

Specific changes include:

1) removed 161_fix_system_metadata_none_strings.py. Nothing to
do here since this was a data conversion (no schema changes)

2) Removed some placeholder migrations for the Grizzly stable
branch.

Change-Id: I94a080ca122af009304975044f33c5e5192834ab
Implements: blueprint compact-havana-db-migrations
Normalize the weights instead of using raw values

The weight system is being used by the scheduler and the cells code.
Currently this system is using the raw values instead of normalizing them.
This makes difficult to properly use multipliers for establishing the
relative importance between two wheighers (one big magnitude could
shade a smaller one). This change introduces weight normalization so
that:

- From an operator point of view we can prioritize the weighers that
  we are applying. The only way to do this is being sure that all the
  weighers will give a value in a known range, so that it is
  not needed to artificially use a huge multiplier to prioritize a
  weigher.

- From a weigher developer point of view, somebody willing to implement
  one has to care about 1) returning a list of values, 2) setting the
  minimum and maximum values where the weights can range, if they are
  needed and they are significant for the weighing. For a weigher
  developer there are two use cases:

    Case 1: Use of a percentage instead of absolute values (for example, %
    of free RAM). If we compare two nodes focusing on the percentage of free
    ram, the maximum value for the weigher is 100. If we have two nodes one
    with 2048 total/1024 free, and the second one 1024 total/512 free they
    will get both the same weight, since they have the same % of free RAM
    (that is, the 50%).

    Case 2: Use of absolute values. In this case, the maximum of the weigher
    will be the maximum of the values in the list (in the case above, 1024)
    or the maximum value that the magnitude could take (in the case above,
    2048). How this maximum is set, is a decision of the developer. He may
    let the operator choose the behaviour of the weigher though.

- From the point of view of the scheduler we ensure that it is using
  normalized values, and not leveraging the normalization mechanism to the
  weighers.

Changes introduced this commit:

1) it introduces weight normalization so that we can apply multipliers
   easily. All the weights for an object will be normalized between 0.0 and
   1.0 before being sumed up, so that the final weight for a host will be:

    weight = w1_multiplier * norm(w1) + w2_multiplier * norm(w2) + ...

2) weights.BaseWeigher has been changed into an ABC so that we enforce
   that all weighers have the expected methods.

3) weights.BaseWeigher.weigh_objects() does no longer sum up the
   computer weighs to the object, but it rather returns a list that will be
   then normalized and added to the existing weight by BaseWeightHandler

4) Adapt the existing weighers to the above changes. Namely
    - New 'offset_weight_multiplier' for the cell weigher
      nova.cells.weights.weight_offset.WeightOffsetWeigher
    - Changed the name of the existing multiplier methods.

5) unittests for all of the introduced changes.

Implements blueprint normalize-scheduler-weights

DocImpact: Now weights for an object are normalized before suming them
up. This means that each weigher will take a maximum value of 1. This
may have an impact for operators that are using more than one weigher
(currently there is only one weigher: RAMWeiger) and for operators using
cells (where we have several weighers). It is needed to review then the
multipliers used and adjust them properly in case they have been
modified.

Docimpact: There is a new configuration option 'offset_weight_multiplier'
in nova.cells.weights.weight_offset.WeightOffsetWeigher

Change-Id: I81bf90898d3cb81541f4390596823cc00106eb20
Refactor PXE DHCP Option support.

The previous support imported pxe into driver, but driver is meant to
be entirely decoupled from specific baremetal drivers.

Co-Authored-By: Robert Collins <rbtcollins@hp.com>

Change-Id: Icc7c40fe06b520aa220906122f5f6be8d79a3ac4
VMWare: bug fix for Vim exception handling

The exception handling in the Vim module was too broad and did not treat
the suds.MethodNotFound exception. This would result in a exception
that "__deepcopy__" is not supported in the VIM module.

In the case of an exception the VMwareAPISession class may be deleted.
The patch also ensures that the deletion is able to logout if necessary.

Closes-bug: #1228847

Change-Id: Ic85be9b3407c444db4b3a074108c01d9141c61be
Stop, Rescue, and Delete should give guest a chance to shutdown

Currently in libvirt stop, shelve, rescue, and delete simply
destroy the underlying VM. Some GuestOS's do not react well to this
type of power failure, and so it would be better if these operations
followed the same approach as soft_reboot and give the guest as
chance to shutdown gracefully.  Even where VM is being deleted,
it may be booted from a volume which will be reused on another
server.

The change is implemented by adding a clean_shutdown parameter
to the relevant methods from the compute/api layer downwards
and into the virt drivers.  The implementation in the libvirt
driver is also provided.  Other drivers are modified just to
expect the additional parameter.

The timer configuration value previous used by soft_reboot in
libvirt is moved up to virt/driver so it can be used by other drivers.

The timer logic itself is changed from simple loop counting with one
second sleeps to a more precise approach, since testing showed that
other calls in the loop could introduce a difference of up to +50% on
the expected duration. This can extent the default two minute to
three minutes, which would not be acceptable in some environments
and breaks some of the tempest tests.

A separate config value defines what the default shutdown
behaviour for delete should be (default False to keep compatibility
with current behaviour).

This code is structured to enable a subsequent API change to add
clean/forced options to the stop and delete methods

Also as a minor tidy-up moved the repeated definition of
FakeLoopingCall in test_libvirt be common across tests

Partially-Implements: blueprint user-defined-shutdown
Closes-Bug: #1196924

DocImpact

Change-Id: Ie69aa2621cb52d6fefdc19f664247b069d6782ee
Rename instance_type to flavor in nova.network tree

Renames 'instance_type' to 'flavor' in the nova.network tree along with
related tests.

Note that 'instance_type_id' is part of the Instance table in the data
model and is not touched as part of this change.

Part of blueprint flavor-instance-type-dedup

Change-Id: I7319315daa77b7334f6d5509cf87faa71148f8ef
Add flavor access methods to Instance object

This adds a native interface to the Instance object to get, save, and
delete either the main or stashed flavors. This will let us migrate
code away from flavors.py and also provide us a versioned method to
handle conversion of flavors, flavor extra specs, and system_metadata
to the JSON blob that we have planned.

Related to blueprint compute-manager-objects

Change-Id: I13a4903460b05798e9c95b2148fe93e4963d6361
Move restore and rebuild operations to Flavor objects

This makes the compute_api restore and rebuild operations use
Flavor objects for their reservation and record-keeping activities.

Related to blueprint compute-manager-objects

Change-Id: If55294e0e1ce678448ee0e9e11fc7b64b5a73386
Make ec2 use Flavor object

This makes the ec2 API use the Flavor object directly. Note that
it also corrects some old usage of non-object instances along
the way.

Related to blueprint compute-manager-objects

Change-Id: Ic13746e085169b6cc0a25fa815c6894a8020c437
Make Serializer/Conductor able to backlevel objects

This improves handling the situation where nova-api sends newer
objects to an older compute node. In this case, the serializer detects
the IncompatibleObjectVersion error and asks conductor to backlevel it.
This means we call into a new method in conductor with the primitive
object that we can't handle, and basically ask it to hydrate it and
call obj_make_compatible() on our behalf and pass it back to us.

This eliminates the need for API to know whether it's talking to
an older compute and allows a newer conductor to assist older
compute nodes during upgrades to a significant degree.

Related-Bug: #1258256
Change-Id: I4b17c8382619e4f73b83c026bf68549ac67a68a2