Restrict keystone cross-tenant ACLs to IDs

The keystoneauth middleware supports cross-tenant access
control using the syntax <tenant>:<user> in container ACLs,
where <tenant> and <user> may currently be either a unique
id or a name. As a result of the keystone v3 API introducing
domains, names are no longer globally unique and are only
unique within a domain. The use of unqualified tenant and
user names in this ACL syntax is therefore not 'safe' in a
keystone v3 environment.

This patch modifies keystoneauth to restrict cross-tenant
ACL matching to use only ids for accounts that are not in
the default domain. For backwards compatibility,
names will still be matched in ACLs when both the requesting
user and tenant are known to be in the default domain AND the
account's tenant is also in the default domain (the default
domain being the domain to which existing tenants are
migrated).

Accounts existing prior to this patch are assumed to be for
tenants in the default domain. New accounts created using a
v2 token scoped on the tenant are also assumed to be in the
default domain. New accounts created using a v3 token scoped
on the tenant will learn their domain membership from the
token info. New accounts created using any unscoped token,
(i.e. with a reselleradmin role) will have unknown domain
membership and therefore be assumed to NOT be in the default
domain.

Despite this provision for backwards compatibility, names
must no longer be used when setting new ACLs in any account,
including new accounts in the default domain.

This change obviously impacts users accustomed to specifying
cross-tenant ACLs in terms of names, and further work will be
necessary to restore those use cases. Some ideas are
discussed under the bug report. With that caveat, this patch
removes the reported vulnerability when using
swift/keystoneauth with a keystone v3 API.

Note: to observe the new 'restricted' behaviour you will need
to setup keystone user(s) and tenant(s) in a non-default domain
and set auth_version = v3.0 in the auth_token middleware config
section of proxy-server.conf. You may also benefit from the
keystone v3 enabled swiftclient patch under review here:
https://review.openstack.org/#/c/91788/

DocImpact

blueprint keystone-v3-support

Closes-Bug:  #1299146

Change-Id: Ib32df093f7450f704127da77ff06b595f57615cb
Fix delete versioning objects when previous is expired

When deleteing versioned objects proxy will try to restore the previous
copy. The COPY request will fail if the previous version is expired but
not handled by object-expirer.

This patch checks COPY respones on the previous copy, if it's
HTTP_NOT_FOUND(mostly because it's expired) proxy will try to restore
with the version before previous.

Closes-Bug #1308446
Change-Id: I17f049ea3ef62723effae8086ec427f6e151cd9c
moving object validation checks to top of PUT method

This adds a sanity check on x-delete headers as
part of check_object_creation method

Change-Id: If5069469e433189235b1178ea203b5c8a926f553
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Extra unit tests for check_delete_headers

A few extra tests to verify check_delete_headers in
constraints.py. A little duplication of coverage of existing
proxy/controllers/test_obj.py:TestObjController.test_POST_delete_at
but these tests call the recently refactored function directly,
and also add tests for X-Delete-After taking precedence over
X-Delete-At.

Change-Id: I129cef15a6feac8a60fd4efbb3535d93f0eaab36
Fix FormPOST max_file_size exceeded bug.

When using FormPOST, if the size of the file being posted exceeds
max_file_size, a HTTP 499 was being returned rather than HTTP 400.

Change-Id: I48c781735c66eccde3deb6f9c3c184aee964a4a5
Delete expired objects in slightly smarter way.

When the expirer tries to delete customer objects, if it just walks through the
containers in order the deamon will tend to send DELETEs to the same container
highly concurrently. This will in turn create a lot of asyncs because of all
the concurrent deletes. If the deletes were spread out to multiple containers
it would improve performance and decrease the number of asyncs made.

Change-Id: I3d08118c197b7f18dd7e880bd5664508934ffd24
Treat 404s as 204 on object delete in proxy

This change adds an optional overrides map to _make_request method
in the base Controller class.

  def make_requests(self, req, ring, part, method, path, headers,
                    query_string='', overrides=None)

Which will be passed on the the best_response method. If set and
no quorum it reached, the override map is used to attempt to find
quorum.

The overrides map is in the form:

    { <response>: <override response>, .. }

The ObjectController, in the DELETE method now passes an override map
to make_requests method in the base Controller class in the form of:

    { 404: 204 }

Statuses/responses that have been overridden are used in calculation
of the quorum but never returned to the user. They are replaced by:

    (STATUS, '', '', '')

And left out of the search for best response.

Change-Id: Ibf969eac3a09d67668d5275e808ed626152dd7eb
Closes-Bug: 1318375
Merge "moving object validation checks to top of PUT method"
Merge "Extra unit tests for check_delete_headers"
Add comments to clarify change to www-authenticate test

Trivial patch to tidy-up change to the functional test for
www-authenticate header and add a comment to explain
that multiple header values might be returned.

Change-Id: If62cb3fd9e11450a2be0cec71e80ecb74a959d04
Related-bug: 1368048
Fix RingBuilder._build_max_replicas_by_tier docstring

The current docstring doesn't include zones, and the order of the
entries is not up to date with the current code. Let's fix this.

Change-Id: Ibabd79427b83d9e8c86b2caeb93dee219c8274c0
Merge "Fix FormPOST max_file_size exceeded bug."
Merge "Restrict keystone cross-tenant ACLs to IDs"
Merge "Delete expired objects in slightly smarter way."
Stop using intersphinx

Remove intersphinx from the docs build as it triggers network calls that
occasionally fail, and we don't really use intersphinx (links other
sphinx documents out on the internet)

This also removes the requirement for internet access during docs build.

This can cause docs jobs to fail if the project errors out on
warnings.

Change-Id: I71e941e2a639641a662a163c682eb86d51de42fb
Related-Bug: #1368910
Merge "Add comments to clarify change to www-authenticate test"
Merge "Fix RingBuilder._build_max_replicas_by_tier docstring"
Fix internal link to keystoneauth in documentation

This patch fixes a broken link at the end of the table in
http://docs.openstack.org/developer/swift/logs.html#swift-source

Change-Id: I989173ac93e0f840997333be0d5cec07eb77b304
Merge "Stop using intersphinx"
Merge "Fix internal link to keystoneauth in documentation"