Setup destination disk from virt_disk_size

When running live-migration --block-migrate on a qcow2 backed
VM without cow image, the destination qcow2 file should be created
with the virtual disk size. For raw images, the virt_disk_size
is set to disk_size to ensure that virt_disk_size is always the
size of the disk that should be re-created.

Update unit tests to be more strict and check for sizes to be correct.

Closes-Bug: #1257355

Change-Id: Ie3be46024f06b9f59af92f5e3918a1958386d4f1
Make it possible to override test timeout value

Currently, we are setting the test timeout value globally
in .testr.conf. It could be useful to be able to override
it per test case (e.g. for migrations tests which usually
take much longer to run than any other tests).

Related-Bug: #1216851

Change-Id: Icee09609f76f4439f6d981641be521d8d695a77e
Give migrations tests more time to run

Migrations tests usually take longer to run that most other
tests. Increase the time out value just for migrations tests,
so that we don't hit various time out errors running unit tests
when CI test nodes slow down a bit.

Closes-Bug: #1216851

Change-Id: I48d7c125397cd282d52eb8e1a1ccb699188d8f14
VMware: fix bug when more than one datacenter exists

In the case that there was more than one datacenter defined on the VC,
then spawning an instance would result in an exception. The reason for this
was that the nova compute would not set the correct datacenter for the
selected datastore.

The fix also takes care of the correct folder selection. This too was a
result of not selecting the correct folder for the data center.

The 'fake' configuration was updated to contain an additional data
center with its on datastore.

Closes-Bug: #1180044
Closes-Bug: #1214850

Co-authored-by: Shawn Harsock <hartsocks@vmware.com>

Change-Id: Ib61811fffcbc80385efc3166c9e366fdaa6432bd
XenAPI: Speedup host_ref cannot change - get it once

The host we run on is fixed - getting this each time from XAPI
is expensive

Change-Id: Ifee9a778f3ed855ef52b001de3fc2ebbd0b972cf
Compact pre-Icehouse database migrations <= 170.

Compacts the pre-Icehouse database migrations <= 170 into a
single migration (170_havana.py).

Pre-Icehouse users will now need to upgrade to Havana before
running any Icehouse migrations.

Tested on MySQL and PostgreSQL schemas on Fedora 19 to ensure the
new compacted migration generates an identical schema to the
previous migrate scripts in Havana.

Specific changes include:

1) removed 161_fix_system_metadata_none_strings.py. Nothing to
do here since this was a data conversion (no schema changes)

2) Removed some placeholder migrations for the Grizzly stable
branch.

Change-Id: I94a080ca122af009304975044f33c5e5192834ab
Implements: blueprint compact-havana-db-migrations
Normalize the weights instead of using raw values

The weight system is being used by the scheduler and the cells code.
Currently this system is using the raw values instead of normalizing them.
This makes difficult to properly use multipliers for establishing the
relative importance between two wheighers (one big magnitude could
shade a smaller one). This change introduces weight normalization so
that:

- From an operator point of view we can prioritize the weighers that
  we are applying. The only way to do this is being sure that all the
  weighers will give a value in a known range, so that it is
  not needed to artificially use a huge multiplier to prioritize a
  weigher.

- From a weigher developer point of view, somebody willing to implement
  one has to care about 1) returning a list of values, 2) setting the
  minimum and maximum values where the weights can range, if they are
  needed and they are significant for the weighing. For a weigher
  developer there are two use cases:

    Case 1: Use of a percentage instead of absolute values (for example, %
    of free RAM). If we compare two nodes focusing on the percentage of free
    ram, the maximum value for the weigher is 100. If we have two nodes one
    with 2048 total/1024 free, and the second one 1024 total/512 free they
    will get both the same weight, since they have the same % of free RAM
    (that is, the 50%).

    Case 2: Use of absolute values. In this case, the maximum of the weigher
    will be the maximum of the values in the list (in the case above, 1024)
    or the maximum value that the magnitude could take (in the case above,
    2048). How this maximum is set, is a decision of the developer. He may
    let the operator choose the behaviour of the weigher though.

- From the point of view of the scheduler we ensure that it is using
  normalized values, and not leveraging the normalization mechanism to the
  weighers.

Changes introduced this commit:

1) it introduces weight normalization so that we can apply multipliers
   easily. All the weights for an object will be normalized between 0.0 and
   1.0 before being sumed up, so that the final weight for a host will be:

    weight = w1_multiplier * norm(w1) + w2_multiplier * norm(w2) + ...

2) weights.BaseWeigher has been changed into an ABC so that we enforce
   that all weighers have the expected methods.

3) weights.BaseWeigher.weigh_objects() does no longer sum up the
   computer weighs to the object, but it rather returns a list that will be
   then normalized and added to the existing weight by BaseWeightHandler

4) Adapt the existing weighers to the above changes. Namely
    - New 'offset_weight_multiplier' for the cell weigher
      nova.cells.weights.weight_offset.WeightOffsetWeigher
    - Changed the name of the existing multiplier methods.

5) unittests for all of the introduced changes.

Implements blueprint normalize-scheduler-weights

DocImpact: Now weights for an object are normalized before suming them
up. This means that each weigher will take a maximum value of 1. This
may have an impact for operators that are using more than one weigher
(currently there is only one weigher: RAMWeiger) and for operators using
cells (where we have several weighers). It is needed to review then the
multipliers used and adjust them properly in case they have been
modified.

Docimpact: There is a new configuration option 'offset_weight_multiplier'
in nova.cells.weights.weight_offset.WeightOffsetWeigher

Change-Id: I81bf90898d3cb81541f4390596823cc00106eb20
Rename instance_type to flavor in nova.network tree

Renames 'instance_type' to 'flavor' in the nova.network tree along with
related tests.

Note that 'instance_type_id' is part of the Instance table in the data
model and is not touched as part of this change.

Part of blueprint flavor-instance-type-dedup

Change-Id: I7319315daa77b7334f6d5509cf87faa71148f8ef
Add flavor access methods to Instance object

This adds a native interface to the Instance object to get, save, and
delete either the main or stashed flavors. This will let us migrate
code away from flavors.py and also provide us a versioned method to
handle conversion of flavors, flavor extra specs, and system_metadata
to the JSON blob that we have planned.

Related to blueprint compute-manager-objects

Change-Id: I13a4903460b05798e9c95b2148fe93e4963d6361
Move restore and rebuild operations to Flavor objects

This makes the compute_api restore and rebuild operations use
Flavor objects for their reservation and record-keeping activities.

Related to blueprint compute-manager-objects

Change-Id: If55294e0e1ce678448ee0e9e11fc7b64b5a73386
Make ec2 use Flavor object

This makes the ec2 API use the Flavor object directly. Note that
it also corrects some old usage of non-object instances along
the way.

Related to blueprint compute-manager-objects

Change-Id: Ic13746e085169b6cc0a25fa815c6894a8020c437
Make Serializer/Conductor able to backlevel objects

This improves handling the situation where nova-api sends newer
objects to an older compute node. In this case, the serializer detects
the IncompatibleObjectVersion error and asks conductor to backlevel it.
This means we call into a new method in conductor with the primitive
object that we can't handle, and basically ask it to hydrate it and
call obj_make_compatible() on our behalf and pass it back to us.

This eliminates the need for API to know whether it's talking to
an older compute and allows a newer conductor to assist older
compute nodes during upgrades to a significant degree.

Related-Bug: #1258256
Change-Id: I4b17c8382619e4f73b83c026bf68549ac67a68a2
api: handle NotImplementedError for console output

Some drivers do not implement getting console output.  Catch this
exception at the API layer and return a 501 from the API in this case.

Change-Id: I144850f4431a7588430da298ba348039bed25403
List NotImplementedError as client exception

Mark NotImplementedError as a client exception for the
get_console_output rpc api method.  This is an expected result if the
driver does not implement this functionality.  If raised, the rpc layer
should not log a traceback.

Change-Id: I60747bf48c378053eb0d36337fcc26201b4840f9
hyperv: remove get_console_output method

The get_console_method() in the hyperv driver only logged a debug
message and returned an empty string.  It seems like much better and
expected behavior to not implement the method at all so that a
NotImplementedError is raised.  This will properly propagate back
through the API.  Not Implemented is certainly different to the user
than telling them their console output is always empty.

Change-Id: I9df81a74ac533a73f30305f3eb12a8ff7608b293
vmwareapi: raise on get_console_output

Commit 2ce3461595a490c9a02272ee4a1ab1eb55c27f60 updated
get_console_output() in the vmwareapi driver to return a static string.
Since this isn't supported, I think the most correct thing to do here is
to raise NotImplementedError.  That will get properly propogated back
through the API.

The original change was concerned with getting a friendly message to the
user.  That should be done by the consumer of the API.  For example, if
Horizon got that as an error back, it could present a special message to
the user indicating that the operation is not supported for that
instance.

Change-Id: Ia17330b9084bf3e82219a0c143485174c3e8f6db
Fix race in unit tests, which can cause gate job to fail

The FakeLibvirtTests.test_openAuth_can_refuse_None_uri test changes the value
of the global variable allow_default_uri_connection. We need to clean up this
change up after running this test so as we don't effect other tests that
expect the value of this variable to be the default.

In particular this test was running before the
LibvirtNonblockingTestCase.test_tpool_execute_calls_libvirt which ran when
the value was changed causing this test to fail. This only occurred when we
the test_libvirt used the fake libvirt driver, which happens when the tests
run inside a virtualenv environment.

Change-Id: I68c30dffd05daae6c0d5fbc6ec005c0fab4f09be
Closes-Bug: 1258553
Handle InstanceUserDataMalformed in create server v2 api

The following patch adds missing support for handling
InstanceUserDataMalformed. In addition, it also adds test cases
in both the v2 and v3 api.

Change-Id: I1b474318248f64eed994f67c6d91db38d5579a63
Closes-bug: #1257146
Revert graceful shutdown patch

aka: "Stop, Rescue, and Delete should give guest a chance to shutdown"

This reverts commit b32d01d44ca5711c96d192df51bf7acd34f52556.

Unfortunately this patch had an unintended side effect that we didn't
notice until after it was merged.  It adds about 25 minutes to tempest
runs, which isn't really acceptable in the gate.  We need to revert this
until we can sort that out.

Change-Id: Ibdd5c86e60b5ede4ec1a38d9efa7c94d60de1720
Remove smoketests

We don't use them anymore. Also remove boto_v6 since only smoketests
have was using it.

Change-Id: Iee206fd4ba6cd502ded794f2de302062604d4186