Improved coverage for glance.api.*

Added new tests for glance.api.v2.image_data,
glance.api.v2.image_members, glance.api.v2.images,
glance.api.v2.schemas, glance.api.v2.image_tags,
glance.api.cached_images.

Part of blueprint improve-api-unit-tests

Change-Id: I2bd39e12366dd1b185bf60a8cd1d03a162ba5896
Moving eventlet.hubs.use_hub call up

Under the design concept of eventlet, we need to call
eventlet.hubs.use_hub() before the application starts
doing any I/O. And it makes custom log handlers register
correct event listeners.

Closes-bug: 1247889
Change-Id: I3d43021c8cb6ed630123781f3686d62e5159edfc
Signed-off-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>
Adding status field to image location -- scrubber changes

Adding a status field to image's each location property, each location
status can be 'active', 'pending_delete' and 'deleted'.

Under location's status information Scrubber service can make cleanup
based on DB records also but not a dedicated queue-file for each image.

This is third part of this change which covered scrubber DB queue
enablement and double queue supporting.

Partially-Implements BP: image-location-status

Change-Id: I8ee9e9254c8371bbbce8b28d643e3d474b2361bc
Signed-off-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>
Fixes the failure of updating or deleting image empty property

When property protection feature enabled, end user couldn't uses
Glance v2 API to update or delete a property of an image successfully
which value is empty, client will receive a http 500 error and a
relevant error log could be found in glance-api service log.

Closes-bug: #1332103

Change-Id: I1f9f181cea714e6ba26388d125bb7023e7a14305
Signed-off-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>
Catch exception.InUseByStore at API layer

If user create an image with Ceph backend driver, and then create
an instance based on the image. As a result, user will run into
500 error when deleting the image. Because rbd driver is raising
exception.InUseByStore()at [1] when the image is in use, but no
code to catch the exception at REST API layer.

[1] https://github.com/openstack/glance/blob/master/glance/store/rbd.py#L290

Closes-Bug: 1345942

Change-Id: I350c235944aab226dc96de169be2087e2b599ab4
Enable F821 check: undefined name 'name'

Enable F821 and fix usage of undefined variables.

In glance/common/client.py in BaseClient class image_iterator()
function returns instances of unexisting classes:
- SendFileIterator
- ImageBodyIterator
Since we have not these classes functionality now, we will use
utils.chunkreadable() function instead.

Delete image_iterator() function.

Change-Id: Ie1a8c66e5df2836e6e9aadfb1013d69fc6f710e6
Decouple read chunk size from write chunk size

When using the copy_from option, readers and writers can have different
speeds to respectively read and write.
A reader timeout will happen if the writer is slow and the writer is
being asked to write a lot. This is currently happening when using
the VMware store and copying from an HTTP server. The reader is reading
16MB which takes too long to upload to vCenter which is causing a
timeout from the HTTP server. The writer should be able to control the
size of the chunks being read when using copy_from: this way the writer
will write fast enough to not make the reader timeout.

This patch addresses the issue by introducing the notion of read chunk
size and write chunk size. Each store can have its own value for read
and write. The write chunk size of the destination store will be used
as the read chunk size of the source store in case of an image-create
where the copy_from option is specified.

Closes-Bug: #1336168

Signed-off-by: Arnaud Legendre <alegendre@vmware.com>
Signed-off-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>

Change-Id: I4e0c563b8f3a5ced8f65fcca83d341a97729a5d4
Log upload failed exception trace rather than debug

We're seeing race failures in test_register_http_image
in the gate due to a 500 coming back from the backing
store when uploading the image in the v1 API, but it's
logged at debug level so we can't fingerprint on the
failure in logstash.

This changes bumps the logging of the error from debug
to exception so we get the trace on the error also.

If it turns out that we have expected exception cases
in the future and tracing on those isn't necessary,
we can filter them out in a separate change.

Change-Id: I87d1f8ae584b41456eae4f19a0bc60f13f47b5ac
Related-Bug: #1347692
Use `_LW` where appropriate in db/sqla/api

Moved some of the sqlalchemy driver code to use the `i18n._*`
translation functions.

Change-Id: Ib78dbf435982289eae45014876c4da6258631c05
Updated from global requirements

Change-Id: Ie31a0772e62e6522186cf6f8bf8b8f032713786d
Merge "Use `_LW` where appropriate in db/sqla/api"
Merge "Moving eventlet.hubs.use_hub call up"
Do not log password in swift URLs in g-registry

There was a debug level log with the locations added to it.
This change fixes the log to not contain that sensitive info.

Fixes bug 1348838

Change-Id: I3c5b29616c8d76bed17dbd31a8f4fc7ccd2dd945
Quota column name 'key' in downgrade script

006_mysql_downgrade didn't correctly quote 'key' column of
'image_properties' table when creating index, the index wasn't created
as needed, and then operator could not run any upgrade script
successfully since 006_mysql_upgrade script could not to handle this
unexpected db status.

Note, this issue doesn't impact one-way upgrade migration operation,
since bug existed in downgrade script. But once operator do a
reciprocating migration operation, this issue will rise up as a blocker.

Closes-bug: 1351413

Change-Id: Ie1ed1b7f06b8a0aeede70c3d4dd061c857e448da
Signed-off-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>
Trivial fix typo: Unavilable to Unavailable

This patch fixes a typo in an exception message.

Change-Id: I204b0bff94970d8743872cd184b8431d30dd1341
Merge "Fixes the failure of updating or deleting image empty property"
Merge "Do not log password in swift URLs in g-registry"
Merge "Trivial fix typo: Unavilable to Unavailable"
Merge "Enable F821 check: undefined name 'name'"
Fix RBD store to use READ_CHUNKSIZE

Change 4e0c563b8f3a5ced8f65fcca83d341a97729a5d4 was incomplete and
missed a variable in the rbd store.

This patch fixes the issue by using store.READ_CHUNKSIZE instead of
store.chunk_size

Change-Id: I2dc5b1514d3e86ec129c1fc88de096d98a64f806
Closes-Bug: #1352509