Fix a lazy-load exception in security_group_update()

When security_group_api is nova, the update_security_group api raises http
500 error: group rules fails with lazy load exception. Because the rule
infos of the security group will be return in response body, but the rule
attribute hasn't been preloaded.

Closes-Bug: #1244018
Change-Id: Ie07879ac22e4add90f75968778bf334915e8349c
Setup destination disk from virt_disk_size

When running live-migration --block-migrate on a qcow2 backed
VM without cow image, the destination qcow2 file should be created
with the virtual disk size. For raw images, the virt_disk_size
is set to disk_size to ensure that virt_disk_size is always the
size of the disk that should be re-created.

Update unit tests to be more strict and check for sizes to be correct.

Closes-Bug: #1257355

Change-Id: Ie3be46024f06b9f59af92f5e3918a1958386d4f1
Make it possible to override test timeout value

Currently, we are setting the test timeout value globally
in .testr.conf. It could be useful to be able to override
it per test case (e.g. for migrations tests which usually
take much longer to run than any other tests).

Related-Bug: #1216851

Change-Id: Icee09609f76f4439f6d981641be521d8d695a77e
Give migrations tests more time to run

Migrations tests usually take longer to run that most other
tests. Increase the time out value just for migrations tests,
so that we don't hit various time out errors running unit tests
when CI test nodes slow down a bit.

Closes-Bug: #1216851

Change-Id: I48d7c125397cd282d52eb8e1a1ccb699188d8f14
VMware: fix bug when more than one datacenter exists

In the case that there was more than one datacenter defined on the VC,
then spawning an instance would result in an exception. The reason for this
was that the nova compute would not set the correct datacenter for the
selected datastore.

The fix also takes care of the correct folder selection. This too was a
result of not selecting the correct folder for the data center.

The 'fake' configuration was updated to contain an additional data
center with its on datastore.

Closes-Bug: #1180044
Closes-Bug: #1214850

Co-authored-by: Shawn Harsock <hartsocks@vmware.com>

Change-Id: Ib61811fffcbc80385efc3166c9e366fdaa6432bd
XenAPI: Speedup host_ref cannot change - get it once

The host we run on is fixed - getting this each time from XAPI
is expensive

Change-Id: Ifee9a778f3ed855ef52b001de3fc2ebbd0b972cf
Compact pre-Icehouse database migrations <= 170.

Compacts the pre-Icehouse database migrations <= 170 into a
single migration (170_havana.py).

Pre-Icehouse users will now need to upgrade to Havana before
running any Icehouse migrations.

Tested on MySQL and PostgreSQL schemas on Fedora 19 to ensure the
new compacted migration generates an identical schema to the
previous migrate scripts in Havana.

Specific changes include:

1) removed 161_fix_system_metadata_none_strings.py. Nothing to
do here since this was a data conversion (no schema changes)

2) Removed some placeholder migrations for the Grizzly stable
branch.

Change-Id: I94a080ca122af009304975044f33c5e5192834ab
Implements: blueprint compact-havana-db-migrations
Normalize the weights instead of using raw values

The weight system is being used by the scheduler and the cells code.
Currently this system is using the raw values instead of normalizing them.
This makes difficult to properly use multipliers for establishing the
relative importance between two wheighers (one big magnitude could
shade a smaller one). This change introduces weight normalization so
that:

- From an operator point of view we can prioritize the weighers that
  we are applying. The only way to do this is being sure that all the
  weighers will give a value in a known range, so that it is
  not needed to artificially use a huge multiplier to prioritize a
  weigher.

- From a weigher developer point of view, somebody willing to implement
  one has to care about 1) returning a list of values, 2) setting the
  minimum and maximum values where the weights can range, if they are
  needed and they are significant for the weighing. For a weigher
  developer there are two use cases:

    Case 1: Use of a percentage instead of absolute values (for example, %
    of free RAM). If we compare two nodes focusing on the percentage of free
    ram, the maximum value for the weigher is 100. If we have two nodes one
    with 2048 total/1024 free, and the second one 1024 total/512 free they
    will get both the same weight, since they have the same % of free RAM
    (that is, the 50%).

    Case 2: Use of absolute values. In this case, the maximum of the weigher
    will be the maximum of the values in the list (in the case above, 1024)
    or the maximum value that the magnitude could take (in the case above,
    2048). How this maximum is set, is a decision of the developer. He may
    let the operator choose the behaviour of the weigher though.

- From the point of view of the scheduler we ensure that it is using
  normalized values, and not leveraging the normalization mechanism to the
  weighers.

Changes introduced this commit:

1) it introduces weight normalization so that we can apply multipliers
   easily. All the weights for an object will be normalized between 0.0 and
   1.0 before being sumed up, so that the final weight for a host will be:

    weight = w1_multiplier * norm(w1) + w2_multiplier * norm(w2) + ...

2) weights.BaseWeigher has been changed into an ABC so that we enforce
   that all weighers have the expected methods.

3) weights.BaseWeigher.weigh_objects() does no longer sum up the
   computer weighs to the object, but it rather returns a list that will be
   then normalized and added to the existing weight by BaseWeightHandler

4) Adapt the existing weighers to the above changes. Namely
    - New 'offset_weight_multiplier' for the cell weigher
      nova.cells.weights.weight_offset.WeightOffsetWeigher
    - Changed the name of the existing multiplier methods.

5) unittests for all of the introduced changes.

Implements blueprint normalize-scheduler-weights

DocImpact: Now weights for an object are normalized before suming them
up. This means that each weigher will take a maximum value of 1. This
may have an impact for operators that are using more than one weigher
(currently there is only one weigher: RAMWeiger) and for operators using
cells (where we have several weighers). It is needed to review then the
multipliers used and adjust them properly in case they have been
modified.

Docimpact: There is a new configuration option 'offset_weight_multiplier'
in nova.cells.weights.weight_offset.WeightOffsetWeigher

Change-Id: I81bf90898d3cb81541f4390596823cc00106eb20
Refactor PXE DHCP Option support.

The previous support imported pxe into driver, but driver is meant to
be entirely decoupled from specific baremetal drivers.

Co-Authored-By: Robert Collins <rbtcollins@hp.com>

Change-Id: Icc7c40fe06b520aa220906122f5f6be8d79a3ac4
VMWare: bug fix for Vim exception handling

The exception handling in the Vim module was too broad and did not treat
the suds.MethodNotFound exception. This would result in a exception
that "__deepcopy__" is not supported in the VIM module.

In the case of an exception the VMwareAPISession class may be deleted.
The patch also ensures that the deletion is able to logout if necessary.

Closes-bug: #1228847

Change-Id: Ic85be9b3407c444db4b3a074108c01d9141c61be
Stop, Rescue, and Delete should give guest a chance to shutdown

Currently in libvirt stop, shelve, rescue, and delete simply
destroy the underlying VM. Some GuestOS's do not react well to this
type of power failure, and so it would be better if these operations
followed the same approach as soft_reboot and give the guest as
chance to shutdown gracefully.  Even where VM is being deleted,
it may be booted from a volume which will be reused on another
server.

The change is implemented by adding a clean_shutdown parameter
to the relevant methods from the compute/api layer downwards
and into the virt drivers.  The implementation in the libvirt
driver is also provided.  Other drivers are modified just to
expect the additional parameter.

The timer configuration value previous used by soft_reboot in
libvirt is moved up to virt/driver so it can be used by other drivers.

The timer logic itself is changed from simple loop counting with one
second sleeps to a more precise approach, since testing showed that
other calls in the loop could introduce a difference of up to +50% on
the expected duration. This can extent the default two minute to
three minutes, which would not be acceptable in some environments
and breaks some of the tempest tests.

A separate config value defines what the default shutdown
behaviour for delete should be (default False to keep compatibility
with current behaviour).

This code is structured to enable a subsequent API change to add
clean/forced options to the stop and delete methods

Also as a minor tidy-up moved the repeated definition of
FakeLoopingCall in test_libvirt be common across tests

Partially-Implements: blueprint user-defined-shutdown
Closes-Bug: #1196924

DocImpact

Change-Id: Ie69aa2621cb52d6fefdc19f664247b069d6782ee
Rename instance_type to flavor in nova.network tree

Renames 'instance_type' to 'flavor' in the nova.network tree along with
related tests.

Note that 'instance_type_id' is part of the Instance table in the data
model and is not touched as part of this change.

Part of blueprint flavor-instance-type-dedup

Change-Id: I7319315daa77b7334f6d5509cf87faa71148f8ef
Add flavor access methods to Instance object

This adds a native interface to the Instance object to get, save, and
delete either the main or stashed flavors. This will let us migrate
code away from flavors.py and also provide us a versioned method to
handle conversion of flavors, flavor extra specs, and system_metadata
to the JSON blob that we have planned.

Related to blueprint compute-manager-objects

Change-Id: I13a4903460b05798e9c95b2148fe93e4963d6361
Move restore and rebuild operations to Flavor objects

This makes the compute_api restore and rebuild operations use
Flavor objects for their reservation and record-keeping activities.

Related to blueprint compute-manager-objects

Change-Id: If55294e0e1ce678448ee0e9e11fc7b64b5a73386
Make ec2 use Flavor object

This makes the ec2 API use the Flavor object directly. Note that
it also corrects some old usage of non-object instances along
the way.

Related to blueprint compute-manager-objects

Change-Id: Ic13746e085169b6cc0a25fa815c6894a8020c437
Make Serializer/Conductor able to backlevel objects

This improves handling the situation where nova-api sends newer
objects to an older compute node. In this case, the serializer detects
the IncompatibleObjectVersion error and asks conductor to backlevel it.
This means we call into a new method in conductor with the primitive
object that we can't handle, and basically ask it to hydrate it and
call obj_make_compatible() on our behalf and pass it back to us.

This eliminates the need for API to know whether it's talking to
an older compute and allows a newer conductor to assist older
compute nodes during upgrades to a significant degree.

Related-Bug: #1258256
Change-Id: I4b17c8382619e4f73b83c026bf68549ac67a68a2
api: handle NotImplementedError for console output

Some drivers do not implement getting console output.  Catch this
exception at the API layer and return a 501 from the API in this case.

Change-Id: I144850f4431a7588430da298ba348039bed25403
List NotImplementedError as client exception

Mark NotImplementedError as a client exception for the
get_console_output rpc api method.  This is an expected result if the
driver does not implement this functionality.  If raised, the rpc layer
should not log a traceback.

Change-Id: I60747bf48c378053eb0d36337fcc26201b4840f9
hyperv: remove get_console_output method

The get_console_method() in the hyperv driver only logged a debug
message and returned an empty string.  It seems like much better and
expected behavior to not implement the method at all so that a
NotImplementedError is raised.  This will properly propagate back
through the API.  Not Implemented is certainly different to the user
than telling them their console output is always empty.

Change-Id: I9df81a74ac533a73f30305f3eb12a8ff7608b293
vmwareapi: raise on get_console_output

Commit 2ce3461595a490c9a02272ee4a1ab1eb55c27f60 updated
get_console_output() in the vmwareapi driver to return a static string.
Since this isn't supported, I think the most correct thing to do here is
to raise NotImplementedError.  That will get properly propogated back
through the API.

The original change was concerned with getting a friendly message to the
user.  That should be done by the consumer of the API.  For example, if
Horizon got that as an error back, it could present a special message to
the user indicating that the operation is not supported for that
instance.

Change-Id: Ia17330b9084bf3e82219a0c143485174c3e8f6db