VMware: Fix memory leaks caused by caches

Using suds objects as keys for the DatastoreBrowser cache is incorrect
because they don't implement __eq__ and __hash__ for the VIM types.
This always results in cache miss and the cache grows with every spawn()
operation.

This patch fix this by using the 'value' property (which is string) of
the MoRef as key.

Closes-Bug: 1316433
(cherry picked from commit bceb3f96b06ff8a048598724494299cf111bcaf8)

Conflicts:

	nova/tests/virt/vmwareapi/test_imagecache.py
	nova/tests/virt/vmwareapi/test_vmops.py

Change-Id: I2bcaf87e733d51055566aee41bb0a7e254027ba9
VMware: Fix fake raising the wrong exception in _remove_file

fake was raising exception.FileNotFound error if a file was not found.
However, an actual delete on a non-existent file raises the error
returned by vSphere, which is error_util.FileNotFoundException. This
breaks tests which exercise error paths expecting the real error to be
raised.

Fortunately this change doesn't break any existing tests. However, I'm
not sure that's a good thing.

This change is required to enable a test in change
I3df3d614656e511c909b6c1837582c0d34bf84c6.

Change-Id: I2025bffa887582eaa9e9072d0400f90ca97d1898
(cherry picked from commit 529b5ab2e195181687c7cc30ece8ddbf160bdeab)
VMware: Fix race in spawn() when resizing cached image

spawn() guards against multiple threads simultaneously attempting to
cache the same image, but it wasn't guarding against them
simultanously trying to create a resized copy in the cache. Attempting
to create a large number of images simultaneously of an uncached image
would result in a race to create the resized image. This resulted in 2
classes of failed instance:

1. Instances whose disk was a linked clone of a copy which had been
   subsequently overwritten. These were corrupt.
2. Instances whose spawn() failed in ExtendVirtualDisk_Task due to a
   locked image.

This patch creates a Nova-local lock for the resized image. The image
is in a per-Nova directory on the datastore, so inter-Nova locking is
not a concern. The lock guards both testing for the existence of the
image, and its creation. Therefore when multiple processes race, only
1 will create the resized copy, and all others will find and use it.
In normal usage this will add the overhead of an additional
uncontended local lock creation and deletion in spawn().

In wrapping this code in a lock, we also make certain that any failure
to create the resized image is appropriately cleaned up. Otherwise
subsequent users will attempt to use a corrupt copy.

Conflicts:
	nova/virt/vmwareapi/vmops.py

Required changes:
  datastore.build_path -> ds_util.build_datastore_path
  No datastore object
  vm_util.copy_virtual_disk -> _copy_virtual_disk
  self.fake_image_uuid -> 'fake_image_uuid'
  removed use of _LE() for log message

This change includes a test which depends on change
I2025bffa887582eaa9e9072d0400f90ca97d1898.

Change-Id: I3df3d614656e511c909b6c1837582c0d34bf84c6
Closes-bug: 1333587
(cherry picked from commit 994cdb234b2b16d97f0276c6356db65817944ee2)
Merge "VMware: Fix memory leaks caused by caches" into stable/icehouse
Merge "VMware: Fix fake raising the wrong exception in _remove_file" into stable/icehouse
Merge "VMware: Fix race in spawn() when resizing cached image" into stable/icehouse
Support aggregate API in cells

Change-Id: I1d687de36787ec70bd14a0ad613b405bcea2938a
Send up AZ and instance_name in system metadata

Change-Id: I63bbc86b4dd4cf107b09f10000b7c9ab06822ad6
Support consoleauth at child cells

Change-Id: I1475e3457fd7d153c90acddb22f103a5abde6f74
Restricted and direct only cell scheduler filters

Change-Id: I38a1a5c1fa90c9ec26ad9184deb0aa60f72abebf
Support flavor create call with cells

Change-Id: I3bfe5a302227e86b9dccac685f89f165abd941b4
Support public-ipv4 metadata lookup without floating IPs

Change-Id: I92dcdd1a5e0bbc0664d97b22c30b9c6990a8b03c
Fix instance AZ check when attaching volumes
Set unique hostnames when launching multiple instances

Change-Id: I5bef77aff24c35dcff1e993034e34cd87e672186
Use fixed_ip for public ip in ec2 responses

Change-Id: Iab568ae7422aa5d96d67fb277294feab9f2d963f
Ensure compute node IP is in list of DHCP servers

This will work for single network installs, and fixes issue
when live migrating the dhcp rule is wrong.

Change-Id: I407b576d9b58df7f34901f039961fdf0032f46e8
Cells: Event-based secgroup/rule/assoc syncing

- ec2 security groups with source groups
- create security group if it doesn't exist on rule create
- management command to force a sync of secgroups down cells

Change-Id: Ibc8c5acb6062e12bc876aff10ec1201b55d228eb
Cells: DB healing for groups/rules/instance assocs

Change-Id: I0721f1eb7f971da4d55cee027c8a4136f6f8b0e7
Healing for EC2 inst/vol and S3 image mappings

Change-Id: Iccb2fdc9b4d2f7a12e4e5569ae6159de50ef5b79
Expire reservations in cells manager

Normally reservations are expired by a periodic task in nova-scheduler.
In a cells deployment quotas are usually enabled only on the api-level
cell. This uses a separate DB from the scheduler, so any reservations
on the top level never get expired.

This change adds the same periodic task to the cells manager so
reservations at all levels have a chance to be expired. The task is
disabled by default and can be turned on with the
CONF.cells.expire_reservations option.

Change-Id: I217f5b5e947662a858eb295f2add5c3227c3b531