Improved coverage for glance.api.*

Added new tests for glance.api.v2.image_data,
glance.api.v2.image_members, glance.api.v2.images,
glance.api.v2.schemas, glance.api.v2.image_tags,
glance.api.cached_images.

Part of blueprint improve-api-unit-tests

Change-Id: I2bd39e12366dd1b185bf60a8cd1d03a162ba5896
Changes HTTP response code for unsupported methods

Requests for resources with an unsupported HTTP method now return a HTTP
response 405 (Method Not Allowed) or 501 (Not Implemented) rather than a
404 (Not Found) for everything.

For example, attempting to DELETE on /v2/images will now return a 405
instead of a 404 and will provide a response header 'Allow' that lists
the valid methods for the resource.

Attempting to use NON_EXISTENT_METHOD on /v2/images will now return a
501.

Attempting to GET on /v2/non_existent_resource will, as expected, return
a 404.

Fixed for v1 and v2.

Change-Id: I5406f8ee423d3d5e66c56a26a7009b4f438a7e0f
Closes-Bug: #1297362
Replacement mox by mock

Replace mox testing library by mock one in:
- glance/tests/unit/common/test_rpc.py,
- glance/tests/unit/test_filesystem_store.py,
- glance/tests/unit/v1/test_registry_client.py,
- glance/tests/unit/v1/test_upload_utils.py,
- glance/tests/unit/v2/test_registry_client.py

Partially Implements: blueprint replace-mox-with-mock

Change-Id: Idbdec5a23d88cee22e9045d43739bfa09b6b4523
Check on schemes not stores

Since the `_upload` method loads the store based on the scheme, we need
to check whether the scheme exists in the known schemes instead of
checking in the list of known stores. Without this fix, stores with a
scheme name different than the store itself will not be usable through
the `x-image-meta-store` header.

Change-Id: Ia33afc1629cb47a31982df7932ac0d9aeb201b4f
Closes-bug: #1357462
Merge "Check on schemes not stores"
Merge "Replacement mox by mock"
Fix collection order issues and unit test failures

A randomized PYTHONHASHSEED caused the glance API to return certain data
in a nondeterministic order, breaking some tests. This patch sorts the
returned data before testing against a pre-sorted expected values.

This patch also fixes some unit test failures with the new version of
testtools (0.9.36) and directly calling setUp(). This was added to this
patch to avoid blocking more Glance tests.

Co-authored-by: Fei Long Wang <flwang@catalyst.net.nz>
Co-authored-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>

Partial-bug: #1348818

Change-Id: Ie92c6bb5663826713ca99e8f2f7d70f3faef6675
Improve Glance profiling

Put osprofiler middleware as a almost first middleware.
This allows us to:
1) Get more preciese duration. We are measuring duration of all middlewares.
2) authenticate middleware is requesting keystone, so if we init osprofiler
   before this middleware, we will get trace infomration from keystone as well.

Change-Id: I11c5d21e6e6b0bd514dee295da410b208d7c798c
Merge "Changes HTTP response code for unsupported methods"
Merge "Improved coverage for glance.api.*"
Merge "Improve Glance profiling"
Fix legacy tests using system policy.json file

Steps to reproduce:

1. Create a new file named '/etc/glance/policy.json' and inside it
paste:

    {"context_is_admin":  "role:admin"}

2. Run tests. Some of the tests under glance.tests.integration
.legacy_functional.test_v1_api will fail with errors such as
'MismatchError: 403 != 201'. These tests will pass without policy.json
in place.

The outcome of the tests should be isolated from the surrounding
environment, so this patch copies policy.json from glance's data files
(glance/tests/etc/policy.json) into the tests directory to avoid this
problem.

Change-Id: I27903ee4ebb7bad243c4f1b2d01096bfb93f1008
Closes-Bug: 1184056
Merge "Fix legacy tests using system policy.json file"
Syncing changes from oslo-incubator policy engine

In this patch, I have copied required changes from oslo-incubator
policy module from commit 0da5de6b548d34300a75f80bf87d6a809c609d2f.

This changes are required for implementation of blueprint
restrict-downloading-images-protected-properties.

blueprint: restrict-downloading-images-protected-properties
Change-Id: I00653411cd0a92503a3096c0dde07e1d6a229353
Restrict users from downloading protected image

Added new rule in policy.json and applied that rule to
'download_image' policy.

For example,
"restricted": "not ('test_key':(test_key)s and role:_member_)"
"download_image": "role:admin or rule:restricted"

So if 'download_image' policy is enforced then in above case only admin or
user who satisfies rule 'restricted' will able to download image. Other users
will not be able to download the image and will get 403 Forbidden response.

In addition, delete property access should be restricted for other users
so that they will not be able to delete the property of the image.

[test_key]
create = admin,member
read = admin,member,_member_
update = admin,member
delete = admin,member

Added new method to create dictionary-like mashup of image core and custom
properties.
Modified v1 and v2 api to add download restriction.
Modified logic of caching to restrict download for v1 and v2 api.

DocImpact:
Need to add new rule in policy.json
"restricted": "not ('test_key':%(test_key)s and role:_member_)"

blueprint: restrict-downloading-images-protected-properties

Change-Id: I05bad0441952150bd15b831ac1b1a0bb9ae79c74
Glance Metadata Definitions Catalog - DB

Implements: blueprint metadata-schema-catalog

A common API hosted by the Glance service for vendors, admins,
services, and users to meaningfully define available key / value
pair and tag metadata. The intent is to enable better metadata
collaboration across artifacts, services, and projects for
OpenStack users.

This is about the definition of the available metadata that can
be used on different types of resources (images, artifacts,
volumes, flavors, aggregates, etc). A definition includes the
properties type, its key, it's description, and it's constraints.
This catalogue will not store the values for specific instance
properties.

Change-Id: I01cc63df3d3abe383c94cfb54598868b4bb729bb
DocImpact
Co-Authored-By: Lakshmi N Sampath <lakshmi.sampath@hp.com>
Co-Authored-By: Wayne Okuma <wayne.okuma@hp.com>
Co-Authored-By: Travis Tripp <travis.tripp@hp.com>
Co-Authored-By: Pawel Koniszewski <pawel.koniszewski@intel.com>
Co-Authored-By: Michal Jastrzebski <michal.jastrzebski@intel.com>
Co-Authored-By: Michal Dulko <michal.dulko@intel.com>
Glance Metadata Definitions Catalog - Seed

Implements: blueprint metadata-schema-catalog

A common API hosted by the Glance service for vendors, admins,
services, and users to meaningfully define available key / value
pair and tag metadata. The intent is to enable better metadata
collaboration across artifacts, services, and projects for
OpenStack users.

This is about the definition of the available metadata that can
be used on different types of resources (images, artifacts,
volumes, flavors, aggregates, etc). A definition includes the
properties type, its key, it's description, and it's constraints.
This catalogue will not store the values for specific instance
properties.

Change-Id: Ib1c1abf80879fb6dcd5ee30c7d2bc65b0ba720d5
DocImpact
Co-Authored-By: Lakshmi N Sampath <lakshmi.sampath@hp.com>
Co-Authored-By: Wayne Okuma <wayne.okuma@hp.com>
Co-Authored-By: Travis Tripp <travis.tripp@hp.com>
Co-Authored-By: Pawel Koniszewski <pawel.koniszewski@intel.com>
Co-Authored-By: Michal Jastrzebski <michal.jastrzebski@intel.com>
Co-Authored-By: Michal Dulko <michal.dulko@intel.com>
Merge "Syncing changes from oslo-incubator policy engine"
Merge "Glance Metadata Definitions Catalog - DB"
Merge "Restrict users from downloading protected image"