Have task/flow logging go to a separate log file

To be able to point the taskflow logging at a task log
we need to be able to separate the logs coming from cinder
from the logs coming from taskflow (as well as the listener
that acts as the intermediary between both projects).

To start the process of having a detailed task log
location, add-in a new log setting that can be used for just
this purpose.

This is not the most ideal way of doing this, since it just
provides a sample logging configuration adjustment, but there
does not appear to be a better way (due to how
cinder/openstack/common/logging.py can't be modified or adjusted
with a better option).

Part of blueprint task-logging

Change-Id: I5e3946c71110ee1b923c2258f3a7eb83c358426b
fix a small typo in development.environment.rst

virutalenv ==> virtualenv

Change-Id: If090ab7b92230fba15d7be98a84b51cce507bedd
Closes-bug: 1358531
Adds CA cert file path option for glance connections

Currently, there is no way to pass CA cert for HTTPS glance
connections. This patch adds new cinder.conf option to enable
secure connection.

Closes-Bug: 1311271
DocImpact - New option added to cinder.conf

Change-Id: I3f39980b94ae314c6351c77f8668ab0922b1c3d8
VMware: Handle exceptions raised by image update

Exceptions raised by glance image update are not handled while writing
to image. This results in volume being stuck in uploading state for
a long time even after image transfer failure. This patch fixes the
exception handling.

Change-Id: I7651c96968caa49c5421d68ee4dbe9e55f7f53b1
Closes-Bug: #1357239
Partial-Bug: #1269345
Create RemoteFSSnapDriver class

Move RemoteFSDriver into its own file.

Refactor pieces needed for qcow2 snapshots into a class.

Also add self.SHARE_FORMAT_REGEX.

Implements bp: remotefs-snaps

Change-Id: I8f9eaed125f4b916b835a63b04239058e34492a9
Ignore No value passed for parameter 'dml' message

There are a lot false positive PyLint errors on
DB migrations for table update method

Change-Id: Id033a1f5af04cde4d8bd888e01bfcc48e656a252
Cache snapshots in request for extension

The extended snapshots extension makes a separate db request for
snapshots which is horribly inefficent. In addition it causes an
issue when using all_tenants=1, because the second request doesn't
pass in the filter. Fix this by caching the snapshot in the request
object like we do for volumes and retrieve the cached object in the
extension.

Change-Id: Ia943438c8d48accb7f62c43f07642c3da2fe34ab
Resolves-bug: #1358960
Modify error code compatible with Mac OS.

Mac OS builtin python errno object has no
attribute 'ENAVAIL'.
Changed
  errno.ENAVAIL --> errno.ENODATA
Another when backend storage service not ready, the
error code "ENAVAIL" change to "ENOPROTOOPT".

Change-Id: I0c41dfe44a4dea02329ff21df881f4c133fe6099
Closes-Bug: 1357010
GlusterFS: Use image_utils for tempfile creation

Besides the aesthetical aspect of replacing ad-hoc code
with an utility function, it's more correct
to perform image conversions in CONF.image_conversion_dir,
which is taken care by the image_utils.temporary_file()
context guard.

Change-Id: Ied5607156f0ab1f1f8cfc889d5b57f3486d6b407
Extending IBMNAS driver to support NFS based GPFS storage system

- Adds support for hardware platform "NAS based on IBM GPFS
  deployments".
- This will enable IBM GPFS-NFS based storage systems to be used
  as a backend in OpenStack environments.

Driver certification test results:
  https://bugs.launchpad.net/cinder/+bug/1358590

DocImpact

Needs an update in ibmnas driver documentation as this change
proposes a new configuration option (ibmnas_platform_type)
to cinder.conf

Change-Id: I09098341ca1948775166883e9b2290061fa6ed37
Implements: blueprint add-gpfs-nas-to-ibmnas
Honor volume:get policy

The fix for bug 1356368 hard-coded a policy check (same as
rule:admin_or_owner) for volume:get.  While in most cases this is
what people want, it'd be good we honor policy setting.

Note that before commit 0505bb268942534ad5d6ecd5e34a4d9b0e7f5c04,
DB query volume_get() actually acted as the policy checker for
volume:get, and it raised VolumeNotFound if context.project_id didn't
match volume['project_id'].  The check_policy() in volume:get didn't
get a chance to raise PolicyNotAuthorized exception.  So in this
change we keep the same behavor.

Change-Id: If43cec5cce977b9220296709b4e243b35b06ecd5
Related-bug: #1356368
Merge "Ignore No value passed for parameter 'dml' message"
Merge "Cache snapshots in request for extension"
Fix variable name in api/v<n>/snapshot.py

Noticed in a recent bug fix that the when the show
method was copy/pasted at some point the variable name
vol was not updated.

Silly annoyance, but thought I'd go ahead and replace vol
with snapshot to make the variable name reflect what's
actually in use here.

Change-Id: I137ecc4987258450e93195cfed5e128063b92914
Integrate OSprofiler and Cinder

*) Add osprofiler wsgi middleware
This middleware is used for 2 things:
1) It checks that person who want to trace is trusted and knows
secret HMAC key.
2) It start tracing in case of proper trace headers
and add first wsgi trace point, with info about HTTP request
*) Add initialization of osprofiler at start of serivce
Set's olso.messaging notifer instance (to send notifications to Ceilometer)

*) Fix fake notifier

NOTE to test this:

You should put to localrc:

  RECLONE=no

  CEILOMETER_NOTIFICATION_TOPICS=notifications,profiler

  ENABLED_SERVICES+=,ceilometer-acompute,ceilometer-acentral,ceilometer-anotification,ceilometer-collector
  ENABLED_SERVICES+=,ceilometer-alarm-evaluator,ceilometer-alarm-notifier
  ENABLED_SERVICES+=,ceilometer-api

You should use python-cinderclient with this patch:

  https://review.openstack.org/#/c/103359/

Run any command with --profile SECRET_KEY

  $ cinder --profile SECRET_KEY create 1
  # it will print <Trace ID>

Get pretty HTML with traces:

  $ osprofiler trace show --html <Profile ID>

note that osprofiler should be run from admin user name & tenant.

Change-Id: Ic3ce587946eaa7e6b19cf35a2cc905ae5de29dd3
Merge "Honor volume:get policy"
Allow backup-to-swift to take swift URL from service catalogue

Since the swift URL is usually in the service catalogue, there is
no need to configure it explicitly.

It can still be set explicitly where manual override is desired.

Change-Id: Ia6c08289b532faed3f6e718d4bbcc1f91978db21
Merge "Fix variable name in api/v<n>/snapshot.py"
Revert test_rootwrap_filter to avoid python2.6 test failure

With oslo.rootwrap 1.3.0.0a1 which has a bug #1340792, that some filter
rules are dependent on its evaluation order, test_rootwrap_filter randomly
fail for a patch adding to etc/cinder/rootwrap.d/volume.filters with
python2.6, as evaluation order is randomly changes.

The bug in oslo.rootwrap is fixed in master branch, but is not yet
released and not available in CI. So this until new oslo.rootwrap is
released, this patch reverts the unit test to avoid to block reviews.

Change-Id: If86b6689ae36aa8810aad5b694ed43dfec1fd0c6
Merge "Modify error code compatible with Mac OS."