Restrict keystone cross-tenant ACLs to IDs

The keystoneauth middleware supports cross-tenant access
control using the syntax <tenant>:<user> in container ACLs,
where <tenant> and <user> may currently be either a unique
id or a name. As a result of the keystone v3 API introducing
domains, names are no longer globally unique and are only
unique within a domain. The use of unqualified tenant and
user names in this ACL syntax is therefore not 'safe' in a
keystone v3 environment.

This patch modifies keystoneauth to restrict cross-tenant
ACL matching to use only ids for accounts that are not in
the default domain. For backwards compatibility,
names will still be matched in ACLs when both the requesting
user and tenant are known to be in the default domain AND the
account's tenant is also in the default domain (the default
domain being the domain to which existing tenants are
migrated).

Accounts existing prior to this patch are assumed to be for
tenants in the default domain. New accounts created using a
v2 token scoped on the tenant are also assumed to be in the
default domain. New accounts created using a v3 token scoped
on the tenant will learn their domain membership from the
token info. New accounts created using any unscoped token,
(i.e. with a reselleradmin role) will have unknown domain
membership and therefore be assumed to NOT be in the default
domain.

Despite this provision for backwards compatibility, names
must no longer be used when setting new ACLs in any account,
including new accounts in the default domain.

This change obviously impacts users accustomed to specifying
cross-tenant ACLs in terms of names, and further work will be
necessary to restore those use cases. Some ideas are
discussed under the bug report. With that caveat, this patch
removes the reported vulnerability when using
swift/keystoneauth with a keystone v3 API.

Note: to observe the new 'restricted' behaviour you will need
to setup keystone user(s) and tenant(s) in a non-default domain
and set auth_version = v3.0 in the auth_token middleware config
section of proxy-server.conf. You may also benefit from the
keystone v3 enabled swiftclient patch under review here:
https://review.openstack.org/#/c/91788/

DocImpact

blueprint keystone-v3-support

Closes-Bug:  #1299146

Change-Id: Ib32df093f7450f704127da77ff06b595f57615cb
account to account copy implementation

Adds ability to copy objects between different accounts (on server side)

Adds new header to `PUT` request:
`X-Copy-From-Account: <account name>`
Account name corresponds to the last part of storage URL.

Adds new header to `COPY` request:
`Destination-Account: <account name>`
Account name corresponds to the last part of storage URL.

If your storage URL is: http://server:8080/v1/AUTH_test
Then the account name is `AUTH_test`

These headers should be used alongside `X-Copy-From` and `Destination` headers
The legacy headers should specify `<container name>/<object name>` path as usual.

DocImpact

Change-Id: I0285fe6a47df9e699ac20ae4a83b0bf23829e1e6
moving object validation checks to top of PUT method

This adds a sanity check on x-delete headers as
part of check_object_creation method

Change-Id: If5069469e433189235b1178ea203b5c8a926f553
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Extra unit tests for check_delete_headers

A few extra tests to verify check_delete_headers in
constraints.py. A little duplication of coverage of existing
proxy/controllers/test_obj.py:TestObjController.test_POST_delete_at
but these tests call the recently refactored function directly,
and also add tests for X-Delete-After taking precedence over
X-Delete-At.

Change-Id: I129cef15a6feac8a60fd4efbb3535d93f0eaab36
make the bind_port config setting required

In a long-term effort to change the recommended ports for Swift,
the first step is to require the bind_port in config files. Later,
we can change the recommended setting.

Anyone currently explicitly setting the ports will not be affected.
Anyone not setting the ports will need to specify them to match their
rings.

DocImpact

Change-Id: Icca83a263acdd0afc9016424a3e9f8c15e944789
Too many if clauses in the _clean_incoming_headers and the
_clean_outgoing_headers routines of the tempurl module

Too many if clauses make code complicated. For more pythonic, you'd
better to use the for ... else clause instead of them.

Fix Bug #1363125

Change-Id: I837235ecb08dd912e56cdfb363c4b4a6cc053e7d
test tempurl header sanitization priority

Change-Id: I0bb3004a717da2f65196bc56b0f7baef49e649e8
Fix FormPOST max_file_size exceeded bug.

When using FormPOST, if the size of the file being posted exceeds
max_file_size, a HTTP 499 was being returned rather than HTTP 400.

Change-Id: I48c781735c66eccde3deb6f9c3c184aee964a4a5
Fix directory value for compile_catalog

Commit 7a192987c0a5edb9e239ffb4aba3b1dac083b41e sets
up swift for translation but the compile_catalog
directory option is pointing at the wrong location
to scan for po files.

Change-Id: Id4dd24ddfde735ef8ef064882bea045361b5db90
Closes-Bug: #1367086
Delete expired objects in slightly smarter way.

When the expirer tries to delete customer objects, if it just walks through the
containers in order the deamon will tend to send DELETEs to the same container
highly concurrently. This will in turn create a lot of asyncs because of all
the concurrent deletes. If the deletes were spread out to multiple containers
it would improve performance and decrease the number of asyncs made.

Change-Id: I3d08118c197b7f18dd7e880bd5664508934ffd24
Merge "Fix directory value for compile_catalog"
Merge "Too many if clauses in the _clean_incoming_headers and the _clean_outgoing_headers routines of the tempurl module"
Merge "account to account copy implementation"
Fix broken www-authenticate functional test

testQuotedWWWAuthenticateHeader functional test started failing
due to a change to keystonemiddleware.auth_token, which now adds
its own www-authenticate header in addition to the one that swift
keystoneauth adds.

This patch changes the functional test to check expected
swift generated header value is in the concatenation of
www-authenticate values.

Verified that functional tests still pass using tempauth.

Closes-Bug: 1368048
Change-Id: I913af077df800a559d259c1622f286ad10eae9df
Merge "make the bind_port config setting required"
Merge "test tempurl header sanitization priority"
Merge "moving object validation checks to top of PUT method"
Merge "Extra unit tests for check_delete_headers"
Add comments to clarify change to www-authenticate test

Trivial patch to tidy-up change to the functional test for
www-authenticate header and add a comment to explain
that multiple header values might be returned.

Change-Id: If62cb3fd9e11450a2be0cec71e80ecb74a959d04
Related-bug: 1368048
Fix RingBuilder._build_max_replicas_by_tier docstring

The current docstring doesn't include zones, and the order of the
entries is not up to date with the current code. Let's fix this.

Change-Id: Ibabd79427b83d9e8c86b2caeb93dee219c8274c0