NetApp NFS efficient clone_image impl

This change implements clone_image funcionality
for NFS drivers. It implements an internal image cache
in nfs shares which will be used to clone images in case
the image_location supplied does not represent an nfs share
which could be used to directly clone image. It also implements
an internal cache cleaning mechanism at regular interval.

blueprint netapp-cinder-nfs-image-cloning

Change-Id: I20eaddf9febc78df33e31bd41b4698b772da3ae7
fix log string in conversion type

By using 'grep -rn "%([a-zA-Z_]*) "', we could find all cases of
'%(variable)'. And we need add an 's' like '%(variable)s'.

Closes-Bug: #1220557
Change-Id: Ib28b7ddc5e5aabe2b82a499d9c242c1982e7f953
Fix bug in Nexenta NFS driver _do_create_volume

Fix bug in _do_create_volume method of Nexenta NFS volume driver.
NMS call nms.folder.create doesn't support third argument, that is why
this call changed to nms.folder.create_with_opts.

Closes-Bug: #1221946
Change-Id: I712ee2919f1bd117e7a1691979639689d95adc92
Fixes pep8 violation in nova

This patch fixes a simple pep8 violation
that nova complains about.

Fixes Bug #1223046

Change-Id: I6f06939e3729b64fcefc5cdb20da05e7d3172be5
Fix Qos Specs association corner case

Fix a corner case in associating QoS Specs with Volumt types that was
not handled well: Raise HTTPBadRequest() to client if associate volume
type, which is already associated one qos specs other than specified
qos specs.  Previously such operation will proceed without error which
results overwriting original association.

Also refactor DB API volume_type_qos_specs_get() to return qos specs
in consistent dict format as other qos specs related APIs.

Fix bug: # 1222692

Change-Id: I9db66d1e3a7469620ba542f5387422685a2d828f
Merge "fix log string in conversion type"
Merge "Fix Qos Specs association corner case"
Fix typo in cinder.tests.test_create_volume_flow

Mock is 3rd party so should be placed after 'time' package.
Changed sheduler -> scheduler

Change-Id: I9c8ef7980c0444a725232d03c29b39d4805586ff
Call to_primitive on volumes.rpcapi.create_volume

cinder.volume.rpcapi.create_volume does not convert the request_spec to
primitive before casting the request. This makes requests containing non
primitive types to fail. For example:

cinder create --metadata=Type=test --source-volid $VOLID 1

This will create a new database record and call create_volume on
volume.rpcapi.  This will fail because VolumeMetadata won't be
serialized correctly when calling cast. This, however, is not True when
--source-volid is not passed because in such case,
scheduler.rpcpai.create_volume will be called, which converts
request_spec to primitive correctly.

Closes-Bug: #1213964
Change-Id: I096d815254c9782390fba05ea9cd9af925635402
Merge "Call to_primitive on volumes.rpcapi.create_volume"
Fixed erroneous force full copy in ceph backup driver

The following steps should not result in a forced full copy:

1. create vol
2. create backup (works fine i.e. uses incremental diff)
3. delete backup
4. create new backup (full copy forced because volume backup-snap exists and is not ignored)

This patch resolves the above problem.

Change-Id: I61c245219f54f7ee942e06e343c5d79d4cab947b
Fixes: bug #1221836
Add check for qemu-img to image_utils fetch_to_raw

Some platforms, particularly PowerPC systems running RHEL,
do not have qemu-img installed by default and do not
support image formats other than RAW.  For these systems,
functions/drivers that use fetch_to_raw() currently fail
because it is assumed that qemu-img will always be installed.

This change updates fetch_to_raw() to function more like the
upload_volume() function which checks the image format and skips
any conversion if the image is already in the RAW format. To
minimize the impact upon platforms that have qemu-img installed
this is being implemented by checking to see if qemu-img is
installed.  If it is installed, no changes are made to the flow
through fetch_to_raw.  If qemu-img is not available and the image
format is not already RAW, an exception is thrown.  Otherwise,
the image is downloaded and we do not progress to the portions of
fetch_to_raw that require qemu-img to be installed.

This commit also updates the test cases that touch this code.

Closes-bug: 1200800

Change-Id: I34540dfa238a8b8e61ff5c30c7e121130bad39b4
Merge "Fix bug in Nexenta NFS driver _do_create_volume"
Merge "Add check for qemu-img to image_utils fetch_to_raw"
fix missing unit in log message

Bad log message:
    "extend volume 1 to 2"
Correct one:
    "extend volume 1 GB to 2 GB"

(also update my profile in .mailmap)

Change-Id: I6d229c4440f2f5443b45a10f3912c20e21c63f32
Move novaclient to requirements.txt

novaclient is a runtime requirement in Havana both for volume
migration and qemu-assisted-snapshots.

Closes-Bug: #1223484
Change-Id: Ie7e8be2d0edf00cdb04f980a6aa247e04807bf67
Fix deleting qos specs key

Previously deleting a key in certain qos specs was accomplished via
'update' API. Unfortunately, 'update' isn't able to tell the
difference between setting a key with no value and deleting a key (and
its value).
This change adds an new API 'delete_keys' to qos_specs API extension.
'delete_keys' API allows client to specify a list of to-be-deleted keys
in one single request (batch mode!), which can be handy when removing
multiple keys in a qos specs.

Example URL and request body for 'delete_keys' API:
 PUT to http://127.0.0.1:8776/v2/qos-specs/QOS_SPECS_UUID/delete_keys
 with body: {'keys': ['foo', 'bar', 'zoo']}

Above example request will result in 'foo', 'bar', 'zoo' key/value
pairs of QOS_SPECS_UUID be marked as deleted in DB.  If QOS_SPECS_UUID
doesn't exist, a 404 error will return; if any key in 'foo', 'bar',
'zoo' couldn't be found in QOS_SPECS_UUID, a 400 error will return with
error message telling which key couldn't be found. Note that Cinder
will puke 400 and stop trying the rest once it encounters a
non-existing (or deleted) key amoung the given list of keys.

This change also fixes 'list'/'show' API includes deleted keys in
result.

Fix bug: # 1223660
Fix bug: # 1223677

Change-Id: Ia3cb07e204d655a9b837b317ce7117feb3c86a2d
Fixing UnicodeEncodeError against volume creating function

When creating a volume with non-English characters and there is
not enough space for it, then an UnicodeEncodeError occurs. So
we need convert any coded exception message to unicode string
for logging.

Fixes bug 1223128

Change-Id: If1d4faad2bcde696f20565deb01226de33caccad
Merge "NetApp NFS efficient clone_image impl"
Merge "Fixed erroneous force full copy in ceph backup driver"