Restrict keystone cross-tenant ACLs to IDs

The keystoneauth middleware supports cross-tenant access
control using the syntax <tenant>:<user> in container ACLs,
where <tenant> and <user> may currently be either a unique
id or a name. As a result of the keystone v3 API introducing
domains, names are no longer globally unique and are only
unique within a domain. The use of unqualified tenant and
user names in this ACL syntax is therefore not 'safe' in a
keystone v3 environment.

This patch modifies keystoneauth to restrict cross-tenant
ACL matching to use only ids for accounts that are not in
the default domain. For backwards compatibility,
names will still be matched in ACLs when both the requesting
user and tenant are known to be in the default domain AND the
account's tenant is also in the default domain (the default
domain being the domain to which existing tenants are
migrated).

Accounts existing prior to this patch are assumed to be for
tenants in the default domain. New accounts created using a
v2 token scoped on the tenant are also assumed to be in the
default domain. New accounts created using a v3 token scoped
on the tenant will learn their domain membership from the
token info. New accounts created using any unscoped token,
(i.e. with a reselleradmin role) will have unknown domain
membership and therefore be assumed to NOT be in the default
domain.

Despite this provision for backwards compatibility, names
must no longer be used when setting new ACLs in any account,
including new accounts in the default domain.

This change obviously impacts users accustomed to specifying
cross-tenant ACLs in terms of names, and further work will be
necessary to restore those use cases. Some ideas are
discussed under the bug report. With that caveat, this patch
removes the reported vulnerability when using
swift/keystoneauth with a keystone v3 API.

Note: to observe the new 'restricted' behaviour you will need
to setup keystone user(s) and tenant(s) in a non-default domain
and set auth_version = v3.0 in the auth_token middleware config
section of proxy-server.conf. You may also benefit from the
keystone v3 enabled swiftclient patch under review here:
https://review.openstack.org/#/c/91788/

DocImpact

blueprint keystone-v3-support

Closes-Bug:  #1299146

Change-Id: Ib32df093f7450f704127da77ff06b595f57615cb
account to account copy implementation

Adds ability to copy objects between different accounts (on server side)

Adds new header to `PUT` request:
`X-Copy-From-Account: <account name>`
Account name corresponds to the last part of storage URL.

Adds new header to `COPY` request:
`Destination-Account: <account name>`
Account name corresponds to the last part of storage URL.

If your storage URL is: http://server:8080/v1/AUTH_test
Then the account name is `AUTH_test`

These headers should be used alongside `X-Copy-From` and `Destination` headers
The legacy headers should specify `<container name>/<object name>` path as usual.

DocImpact

Change-Id: I0285fe6a47df9e699ac20ae4a83b0bf23829e1e6
Fix delete versioning objects when previous is expired

When deleteing versioned objects proxy will try to restore the previous
copy. The COPY request will fail if the previous version is expired but
not handled by object-expirer.

This patch checks COPY respones on the previous copy, if it's
HTTP_NOT_FOUND(mostly because it's expired) proxy will try to restore
with the version before previous.

Closes-Bug #1308446
Change-Id: I17f049ea3ef62723effae8086ec427f6e151cd9c
moving object validation checks to top of PUT method

This adds a sanity check on x-delete headers as
part of check_object_creation method

Change-Id: If5069469e433189235b1178ea203b5c8a926f553
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Extra unit tests for check_delete_headers

A few extra tests to verify check_delete_headers in
constraints.py. A little duplication of coverage of existing
proxy/controllers/test_obj.py:TestObjController.test_POST_delete_at
but these tests call the recently refactored function directly,
and also add tests for X-Delete-After taking precedence over
X-Delete-At.

Change-Id: I129cef15a6feac8a60fd4efbb3535d93f0eaab36
make the bind_port config setting required

In a long-term effort to change the recommended ports for Swift,
the first step is to require the bind_port in config files. Later,
we can change the recommended setting.

Anyone currently explicitly setting the ports will not be affected.
Anyone not setting the ports will need to specify them to match their
rings.

DocImpact

Change-Id: Icca83a263acdd0afc9016424a3e9f8c15e944789
test tempurl header sanitization priority

Change-Id: I0bb3004a717da2f65196bc56b0f7baef49e649e8
Fix FormPOST max_file_size exceeded bug.

When using FormPOST, if the size of the file being posted exceeds
max_file_size, a HTTP 499 was being returned rather than HTTP 400.

Change-Id: I48c781735c66eccde3deb6f9c3c184aee964a4a5
Delete expired objects in slightly smarter way.

When the expirer tries to delete customer objects, if it just walks through the
containers in order the deamon will tend to send DELETEs to the same container
highly concurrently. This will in turn create a lot of asyncs because of all
the concurrent deletes. If the deletes were spread out to multiple containers
it would improve performance and decrease the number of asyncs made.

Change-Id: I3d08118c197b7f18dd7e880bd5664508934ffd24
Treat 404s as 204 on object delete in proxy

This change adds an optional overrides map to _make_request method
in the base Controller class.

  def make_requests(self, req, ring, part, method, path, headers,
                    query_string='', overrides=None)

Which will be passed on the the best_response method. If set and
no quorum it reached, the override map is used to attempt to find
quorum.

The overrides map is in the form:

    { <response>: <override response>, .. }

The ObjectController, in the DELETE method now passes an override map
to make_requests method in the base Controller class in the form of:

    { 404: 204 }

Statuses/responses that have been overridden are used in calculation
of the quorum but never returned to the user. They are replaced by:

    (STATUS, '', '', '')

And left out of the search for best response.

Change-Id: Ibf969eac3a09d67668d5275e808ed626152dd7eb
Closes-Bug: 1318375
Merge "account to account copy implementation"
Fix broken www-authenticate functional test

testQuotedWWWAuthenticateHeader functional test started failing
due to a change to keystonemiddleware.auth_token, which now adds
its own www-authenticate header in addition to the one that swift
keystoneauth adds.

This patch changes the functional test to check expected
swift generated header value is in the concatenation of
www-authenticate values.

Verified that functional tests still pass using tempauth.

Closes-Bug: 1368048
Change-Id: I913af077df800a559d259c1622f286ad10eae9df
Merge "make the bind_port config setting required"
Merge "test tempurl header sanitization priority"
Merge "moving object validation checks to top of PUT method"
Merge "Extra unit tests for check_delete_headers"
Add comments to clarify change to www-authenticate test

Trivial patch to tidy-up change to the functional test for
www-authenticate header and add a comment to explain
that multiple header values might be returned.

Change-Id: If62cb3fd9e11450a2be0cec71e80ecb74a959d04
Related-bug: 1368048
Fix RingBuilder._build_max_replicas_by_tier docstring

The current docstring doesn't include zones, and the order of the
entries is not up to date with the current code. Let's fix this.

Change-Id: Ibabd79427b83d9e8c86b2caeb93dee219c8274c0
Merge "Fix FormPOST max_file_size exceeded bug."
Merge "Restrict keystone cross-tenant ACLs to IDs"