Adding status field to image location -- scrubber changes

Adding a status field to image's each location property, each location
status can be 'active', 'pending_delete' and 'deleted'.

Under location's status information Scrubber service can make cleanup
based on DB records also but not a dedicated queue-file for each image.

This is third part of this change which covered scrubber DB queue
enablement and double queue supporting.

Partially-Implements BP: image-location-status

Change-Id: I8ee9e9254c8371bbbce8b28d643e3d474b2361bc
Signed-off-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>
Catch exception.InUseByStore at API layer

If user create an image with Ceph backend driver, and then create
an instance based on the image. As a result, user will run into
500 error when deleting the image. Because rbd driver is raising
exception.InUseByStore()at [1] when the image is in use, but no
code to catch the exception at REST API layer.

[1] https://github.com/openstack/glance/blob/master/glance/store/rbd.py#L290

Closes-Bug: 1345942

Change-Id: I350c235944aab226dc96de169be2087e2b599ab4
Quota column name 'key' in downgrade script

006_mysql_downgrade didn't correctly quote 'key' column of
'image_properties' table when creating index, the index wasn't created
as needed, and then operator could not run any upgrade script
successfully since 006_mysql_upgrade script could not to handle this
unexpected db status.

Note, this issue doesn't impact one-way upgrade migration operation,
since bug existed in downgrade script. But once operator do a
reciprocating migration operation, this issue will rise up as a blocker.

Closes-bug: 1351413

Change-Id: Ie1ed1b7f06b8a0aeede70c3d4dd061c857e448da
Signed-off-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>
Fix image killed after deletion

If an image upload fails after the image was deleted during
the upload, the image status currently gets set to 'deleted'
then 'killed' once the upload fails. This patch fixes this so
that if the image was deleted it stays in 'deleted' state.

Permitted from_states for transitioning to 'killed' are
'queued' and 'saving'. We now enforce this transition and
raise an exception if it is violated.

Closes-Bug: 1236868
Co-authored-by: Hirofumi Ichihara <ichihara.hirofumi@lab.ntt.co.jp>
Co-authored-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>
Change-Id: I5552220ce264c0d09780bbd0eb7610e7cff0b820
Integrate OSprofiler and Glance

*) Add osprofiler wsgi middleware
This middleware is used for 2 things:
1) It checks that person who want to trace is trusted and knows
secret HMAC key.
2) It start tracing in case of proper trace headers
and add first wsgi trace point, with info about HTTP request

*) Add initialization of osprofiler at start of serivce
Set's olso.messaging notifer instance (to send notifications to
Ceilometer)

*) Add tracing support for sqlalchemy

*) Add profiler CONF group that has to options:
1) Enable or disable profiler
2) Enable or disable sqlalchemy tracing
(the reason why we put this in Conf options, is that usually
db requests create a lot of trace info)

*) Glance-registry wasn't setted properly to send notifications.

To test it you need 2 things:
1) Enable ceilometer in devstack:
http://paste.openstack.org/show/85724/
2) Initialize osprofiler and send out API request. You can use
one of follow approaches:
a. Run this script (pls provide proper credentials)
http://paste.openstack.org/show/85722/
b. Use the function https://review.openstack.org/#/c/111184/
added from glanceclient CLI directly.

DocImpact

Change-Id: I45a19f5eb5304c2b78a9e12cbc0744941a807304
Co-author: Zhi Yan Liu <zhiyanl@cn.ibm.com>#
Fix rally performance job in glance

Recently we merged non backward compability change in rally
https://review.openstack.org/#/c/112045/

So we need to fix plugin

As well change name to something reasonable

Change-Id: I040959e9d1e36f524e88174c9cbbeb7db2f4b66b
Merge "Quota column name 'key' in downgrade script"
Fix migration on older postgres

Since the move to oslo.db, sqlalchemy no longer raises
ProgrammingError.  Instead, it is wrapped in a
oslo.db.exception.DBError. This causes migration to fail on older
postgres, which relies on catching this error to use the correct
constraint name.

Change-Id: Icf7b20eb4e09f3c91aa3df3bcea497f810a6f907
Bump osprofiler requirement to 0.3.0

ofprofiler.notifier.create, which is used by recently merged
commit 94b670c199fcc39784edd90b323b0396914ce66a, does not seem
available in osprofiler 0.1.1.

Closes-Bug: #1354500
Change-Id: I563558efcf4e909e2a97003bf84a0785091fd360
Merge "Fix rally performance job in glance"
Merge "Adding status field to image location -- scrubber changes"
Adding status field to image location -- scrubber queue switching

Adding a status field to image's each location property, each location
status can be 'active', 'pending_delete' and 'deleted'.

Under location's status information Scrubber service can make cleanup
based on DB records also but not a dedicated queue-file for each image.

This is fourth part of this change which covered using scrubber DB queue
in store. This change allow cloud admin/operator deploy a central
Scrubber service(s) on one or two hosts only, before this Glance require
deploy it on each API service host since Scrubber only support load
pending delete jobs from local file.

Partially-Implements BP: image-location-status

Change-Id: I6910b55de8c3b203560d75ff3d1675eda31ae786
Signed-off-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>
Use @mock.patch.object instead of mock.MagicMock

Some unittests replace methods by mock objects, so they are leave mocked
when unittest finished. It can cause a lot of issues, because when we
try to get access to these attributes in the next test, we get mock
object instead of real attribute. Use @mock.patch.object decorator to
get attributes `unmocked` after test.

Change-Id: I23b3926a7671e0669315836da787ec88a812d67b
Merge "Fix image killed after deletion"
Fix bad indentation in glance

Fix for bad indentation warnings from pylint.

Closes-Bug: #1356406

Change-Id: I425de4975a4468495a000e408e8d7cf94b646912
Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
Merge "Adding status field to image location -- scrubber queue switching"
Merge "Use @mock.patch.object instead of mock.MagicMock"
Merge "Fix bad indentation in glance"
Ensure constant order when setting all image tags

The newer version of tox (>=1.7.0) sets a random hash seed, which makes the
order of sets unpredictable. This patchset fixes the test_image_tag_set_all
test, which otherwise broke with a randomized PYTHONHASHSEED. This new tox
behaviour is consistent with python (>=3.3), meaning this change should avoid
bugs on that platform at a later date.

Change-Id: Ie281b04fc9c4c2a96a9eec9bde7b05229ebf57fc
Partial-bug: #1348818
Do not assume order of images

This fixes the test_tag_lifecycle test, which broke with a randomized
PYTHONHASHSEED.

Previously, the test assumed that the values in the list of tag names
would remain constant. The test now sorts the results and compares to a
sorted set.

Change-Id: If5a2ef683fcf55520c7d539f1b90aefaba4739d5
Partial-bug: #1348818