ismount should not raise exceptions

The changes from using os.path.ismount to using
swift.common.utils.ismount has caused problems since the new one
raises exceptions in cases where the old one did not. Daemons have
been encountering this and exiting; servers have been 500ing instead
of 507ing in this case, changing handoff behaviors, etc.

Since the new one was specifically written and tested for this new
behavior, I left that original function as ismount_raw and made
ismount do what it did before.

If there really isn't some reason for this new behavior, I'll be glad
to get rid of ismount_raw and just keep ismount. I couldn't see any
reason for the new behavior myself.

Change-Id: I2b5b17f9ed9656cd8804a5ed568170697d0b183d
Use a tempfile.mkdtemp() based temporary directory

Change-Id: Ie0e8137615348b130b67323d0d2913dc5ebfd5fb
Fix up SAIO to use new gatekeeper

Fix up the SAIO default proxy-server pipeline to use the new
gatekeeper and update to follow the pipeline ordering from proxy
sample.

Change-Id: Iaa4de7abd7fc557cafe6fe40668973e9fc142ca3
Add debug logger on per-test proxy-server

To help debug problems with the proxy-server app setup for each class,
we add a debug logger so that on failure we can see what requests were
actually sent.

Additionally, we move the set_http_connect() call to the same relative
position as the one before it for clarity of intent.

Related bug 1271962

Change-Id: Idc301c06e114b11c358ee4fbc0b2b70ec7743091
Add secondary groups to user during privilege escalation

setgid provides the primary group, setgroups sets the secondary
groups. Prior to this patch, we would do a setgroups with an empty
list, effectively wiping secondary groups. We now verify which
secondary groups the user is member of and escalate the privileges
accordingly.

Change-Id: I33a10edd448b3ac5aa758a8d1d70e582cf421c7d
Closes-Bug: 1269473
Add documentation for GateKeeper

The purpose of GateKeeper mostly relates to the development of new swift code,
so I threw together a guide for development_middleware that covers some basics
with a eye towards metadata handling in-particular.

I also fixed up some missing autodoc's, split out middleware autodoc and added
some ref's here and about so I could link to them from the
development_middleware guide.

DocImpact
Change-Id: I20dd942ea8df9e33c3e794cb49669ffa1332c63e
Merge "Fix up SAIO to use new gatekeeper"
Move test.unit.locale out of gettext's way

When I run unittests on my laptop I get an ImportError from gettext who tries
to "from locale import normalize" and says "ImportError: cannot import name
normalize".  Whit this patch unittests works, not sure why this problem
doesn't show up in my other dev environments.

Also this seems like good form since 'locale' is already a stdlib module.

Change-Id: If0fcd66ce391665b4e4127c610de3246db409a68
Merge "Add debug logger on per-test proxy-server"
Merge "Use a tempfile.mkdtemp() based temporary directory"
Merge "Add documentation for GateKeeper"
Merge "ismount should not raise exceptions"
Add flag to stop swob from always using absolute location.

This is needed for SOS (along with patch
https://github.com/dpgoetz/sos/pull/37)
to work with swift 1.12 . By spec you should always use the absolute
location but this causes a problem with staticweb over a cdn using a
cname. Basically you want to be able to forward the browser to a
relative location instead of whatever full url the proxy server
thinks you are using.

Change-Id: I3fa1d415bf9b566be069458b838f7e65db0c4f39
Add flag to stop swob from always using absolute location.

This is a backport of commit 89deaf5f3cb062cc2c2d7d7d365e84ec2e41a511

This is needed for SOS (along with patch
https://github.com/dpgoetz/sos/pull/37)
to work with swift 1.12 . By spec you should always use the absolute
location but this causes a problem with staticweb over a cdn using a
cname. Basically you want to be able to forward the browser to a
relative location instead of whatever full url the proxy server
thinks you are using.

Change-Id: I28dcbd779d2aa4a6e636c956b058670b8e4b2fd7
Merge "Move test.unit.locale out of gettext's way"
Merge "Add secondary groups to user during privilege escalation"
Merge branch 'milestone-proposed' (1.12.0) into merge_branch

Change-Id: I0e249625c90ef05dddebaf2ada9cdb56c7b7af94
Privileged acct ACL header, new ACL syntax, TempAuth impl.

 * Introduce a new privileged account header: X-Account-Access-Control
 * Introduce JSON-based version 2 ACL syntax -- see below for discussion
 * Implement account ACL authorization in TempAuth

X-Account-Access-Control Header
-------------------------------

Accounts now have a new privileged header to represent ACLs or any other
form of account-level access control.  The value of the header is an opaque
string to be interpreted by the auth system, but it must be a JSON-encoded
dictionary.  A reference implementation is given in TempAuth, with the
knowledge that historically other auth systems often use TempAuth as a
starting point.

The reference implementation describes three levels of account access:
"admin", "read-write", and "read-only".  Adding new access control
features in a future patch (e.g. "write-only" account access) will
automatically be forward- and backward-compatible, due to the JSON
dictionary header format.

The privileged X-Account-Access-Control header may only be read or written
by a user with "swift_owner" status, traditionally the account owner but
now also any user on the "admin" ACL.

Access Levels:

Read-only access is intended to indicate to the auth system that this
list of identities can read everything (except privileged headers) in
the account.  Specifically, a user with read-only account access can get
a list of containers in the account, list the contents of any container,
retrieve any object, and see the (non-privileged) headers of the
account, any container, or any object.

Read-write access is intended to indicate to the auth system that this
list of identities can read or write (or create) any container.  A user
with read-write account access can create new containers, set any
unprivileged container headers, overwrite objects, delete containers,
etc.  A read-write user can NOT set account headers (or perform any
PUT/POST/DELETE requests on the account).

Admin access is intended to indicate to the auth system that this list of
identities has "swift_owner" privileges.  A user with admin account access
can do anything the account owner can, including setting account headers
and any privileged headers -- and thus changing the value of
X-Account-Access-Control and thereby granting read-only, read-write, or
admin access to other users.

The auth system is responsible for making decisions based on this header,
if it chooses to support its use.  Therefore the above access level
descriptions are necessarily advisory only for other auth systems.

When setting the value of the header, callers are urged to use the new
format_acl() method, described below.

New ACL Format
--------------

The account ACLs introduce a new format for ACLs, rather than reusing the
existing format from X-Container-Read/X-Container-Write.  There are several
reasons for this:
 * Container ACL format does not support Unicode
 * Container ACLs have a different structure than account ACLs
  + account ACLs have no concept of referrers or rlistings
  + accounts have additional "admin" access level
  + account access levels are structured as admin > rw > ro, which seems more
    appropriate for how people access accounts, rather than reusing
    container ACLs' orthogonal read and write access

In addition, the container ACL syntax is a bit arbitrary and highly custom,
so instead of parsing additional custom syntax, I'd rather propose a next
version and introduce a means for migration.  The V2 ACL syntax has the
following benefits:
 * JSON is a well-known standard syntax with parsers in all languages
 * no artificial value restrictions (you can grant access to a user named
    ".rlistings" if you want)
 * forward and backward compatibility: you may have extraneous keys, but
    your attempt to parse the header won't raise an exception

I've introduced hooks in parse_acl and format_acl which currently default
to the old V1 syntax but tolerate the V2 syntax and can easily be flipped
to default to V2.  I'm not changing the default or adding code to rewrite
V1 ACLs to V2, because this patch has suffered a lot of scope creep already,
but this seems like a sensible milestone in the migration.

TempAuth Account ACL Implementation
-----------------------------------

As stated above, core Swift is responsible for privileging the
X-Account-Access-Control header (making it only accessible to swift_owners),
for translating it to -sysmeta-* headers to trigger persistence by the
account server, and for including the header in the responses to requests
by privileged users.  Core Swift puts no expectation on the *content* of
this header.  Auth systems (including TempAuth) are responsible for
defining the content of the header and taking action based on it.

In addition to the changes described above, this patch defines a format
to be used by TempAuth for these headers in the common.middleware.acl
module, in the methods format_v2_acl() and parse_v2_acl().  This patch
also teaches TempAuth to take action based on the header contents.  TempAuth
now sets swift_owner=True if the user is on the Admin ACL, authorizes
GET/HEAD/OPTIONS requests if the user is on any ACL, authorizes
PUT/POST/DELETE requests if the user is on the admin or read-write ACL, etc.

Note that the action of setting swift_owner=True triggers core Swift to
add or strip the privileged headers from the responses.  Core Swift (not
the auth system) is responsible for that.

DocImpact: Documentation for the new ACL usage and format appears in
summary form in doc/source/overview_auth.rst, and in more detail in
swift/common/middleware/tempauth.py in the TempAuth class docstring.
I leave it to the Swift doc team to determine whether more is needed.

Change-Id: I836a99eaaa6bb0e92dc03e1ca46a474522e6e826