Restrict keystone cross-tenant ACLs to IDs

The keystoneauth middleware supports cross-tenant access
control using the syntax <tenant>:<user> in container ACLs,
where <tenant> and <user> may currently be either a unique
id or a name. As a result of the keystone v3 API introducing
domains, names are no longer globally unique and are only
unique within a domain. The use of unqualified tenant and
user names in this ACL syntax is therefore not 'safe' in a
keystone v3 environment.

This patch modifies keystoneauth to restrict cross-tenant
ACL matching to use only ids for accounts that are not in
the default domain. For backwards compatibility,
names will still be matched in ACLs when both the requesting
user and tenant are known to be in the default domain AND the
account's tenant is also in the default domain (the default
domain being the domain to which existing tenants are
migrated).

Accounts existing prior to this patch are assumed to be for
tenants in the default domain. New accounts created using a
v2 token scoped on the tenant are also assumed to be in the
default domain. New accounts created using a v3 token scoped
on the tenant will learn their domain membership from the
token info. New accounts created using any unscoped token,
(i.e. with a reselleradmin role) will have unknown domain
membership and therefore be assumed to NOT be in the default
domain.

Despite this provision for backwards compatibility, names
must no longer be used when setting new ACLs in any account,
including new accounts in the default domain.

This change obviously impacts users accustomed to specifying
cross-tenant ACLs in terms of names, and further work will be
necessary to restore those use cases. Some ideas are
discussed under the bug report. With that caveat, this patch
removes the reported vulnerability when using
swift/keystoneauth with a keystone v3 API.

Note: to observe the new 'restricted' behaviour you will need
to setup keystone user(s) and tenant(s) in a non-default domain
and set auth_version = v3.0 in the auth_token middleware config
section of proxy-server.conf. You may also benefit from the
keystone v3 enabled swiftclient patch under review here:
https://review.openstack.org/#/c/91788/

DocImpact

blueprint keystone-v3-support

Closes-Bug:  #1299146

Change-Id: Ib32df093f7450f704127da77ff06b595f57615cb
account to account copy implementation

Adds ability to copy objects between different accounts (on server side)

Adds new header to `PUT` request:
`X-Copy-From-Account: <account name>`
Account name corresponds to the last part of storage URL.

Adds new header to `COPY` request:
`Destination-Account: <account name>`
Account name corresponds to the last part of storage URL.

If your storage URL is: http://server:8080/v1/AUTH_test
Then the account name is `AUTH_test`

These headers should be used alongside `X-Copy-From` and `Destination` headers
The legacy headers should specify `<container name>/<object name>` path as usual.

DocImpact

Change-Id: I0285fe6a47df9e699ac20ae4a83b0bf23829e1e6
Fix delete versioning objects when previous is expired

When deleteing versioned objects proxy will try to restore the previous
copy. The COPY request will fail if the previous version is expired but
not handled by object-expirer.

This patch checks COPY respones on the previous copy, if it's
HTTP_NOT_FOUND(mostly because it's expired) proxy will try to restore
with the version before previous.

Closes-Bug #1308446
Change-Id: I17f049ea3ef62723effae8086ec427f6e151cd9c
Change method _sort_key_for to static

This method does not reference to any attribute of this class, so it's better
to change it to classstatic.

Change-Id: I3ea0ca83cb29ceae9afb20da0c9817a9b044c084
Signed-off-by: Lin Yang <lin.a.yang@intel.com>
Pay attention to all punctual nodes

The proxy sends requests to all storage nodes, but it only needs a
quorum of them to respond before the proxy can, in turn, respond to
the client. Thus, it gets quorum, and then briefly waits to see if the
remainder of the storage nodes respond before giving up on them.

However, the proxy was not paying any attention to the responses from
the non-quorum storage nodes. This would lead to some odd behavior,
like a container PUT where the backends returned (201, 201, 202) would
become a 201 to the client, but the permutation (201, 202, 201) would
become 202. Further, on object PUT, if the last node responded with an
error code, that error wouldn't count towards error-limiting.

The fix is quite simple: after getting quorum, the make_requests()
method was calling a method that returns responses from the remainder
of the nodes, but it was ignoring that return value and making up
responses with dummy values instead. Now, prior to making up dummy
responses, the proxy first uses the responses it already has, and only
fills in dummy responses for nodes that really didn't respond in time.

Change-Id: I0206b6b2272b0e7dcc80fb6c51840d8dae25cee2
moving object validation checks to top of PUT method

This adds a sanity check on x-delete headers as
part of check_object_creation method

Change-Id: If5069469e433189235b1178ea203b5c8a926f553
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Mention storage backends in Associated Projects

Change-Id: I6d88cfe668a557c41e6e6958cd73412ca50c6ba8
Signed-off-by: Prashanth Pai <ppai@redhat.com>
Extra unit tests for check_delete_headers

A few extra tests to verify check_delete_headers in
constraints.py. A little duplication of coverage of existing
proxy/controllers/test_obj.py:TestObjController.test_POST_delete_at
but these tests call the recently refactored function directly,
and also add tests for X-Delete-After taking precedence over
X-Delete-At.

Change-Id: I129cef15a6feac8a60fd4efbb3535d93f0eaab36
make the bind_port config setting required

In a long-term effort to change the recommended ports for Swift,
the first step is to require the bind_port in config files. Later,
we can change the recommended setting.

Anyone currently explicitly setting the ports will not be affected.
Anyone not setting the ports will need to specify them to match their
rings.

DocImpact

Change-Id: Icca83a263acdd0afc9016424a3e9f8c15e944789
Too many if clauses in the _clean_incoming_headers and the
_clean_outgoing_headers routines of the tempurl module

Too many if clauses make code complicated. For more pythonic, you'd
better to use the for ... else clause instead of them.

Fix Bug #1363125

Change-Id: I837235ecb08dd912e56cdfb363c4b4a6cc053e7d
test tempurl header sanitization priority

Change-Id: I0bb3004a717da2f65196bc56b0f7baef49e649e8
Merge "Pay attention to all punctual nodes"
Fixed Typo - dictonary to dictionary

Change-Id: Ia2a07bf1d1a77f6d6af0c8d9f53c2d47c2ba6d75
Fix FormPOST max_file_size exceeded bug.

When using FormPOST, if the size of the file being posted exceeds
max_file_size, a HTTP 499 was being returned rather than HTTP 400.

Change-Id: I48c781735c66eccde3deb6f9c3c184aee964a4a5
Merge "Change method _sort_key_for to static"
Fix directory value for compile_catalog

Commit 7a192987c0a5edb9e239ffb4aba3b1dac083b41e sets
up swift for translation but the compile_catalog
directory option is pointing at the wrong location
to scan for po files.

Change-Id: Id4dd24ddfde735ef8ef064882bea045361b5db90
Closes-Bug: #1367086
Merge "Fixed Typo - dictonary to dictionary"
Merge "Mention storage backends in Associated Projects"
Delete expired objects in slightly smarter way.

When the expirer tries to delete customer objects, if it just walks through the
containers in order the deamon will tend to send DELETEs to the same container
highly concurrently. This will in turn create a lot of asyncs because of all
the concurrent deletes. If the deletes were spread out to multiple containers
it would improve performance and decrease the number of asyncs made.

Change-Id: I3d08118c197b7f18dd7e880bd5664508934ffd24
Merge "Fix directory value for compile_catalog"