Add v2 API to list endpoints middleware

The new API adds better support for storage policies and changes the
response from a list of backend urls to a dictionary with a key
"endpoints" that's a list of of the backend urls and a new key headers
that's a dictionary of headers to send along with the backend request.
In the v2 response format for object requests, there the headers key
includes "X-Backend-Storage-Policy-Index" which indicates the storage
policy index for the endpoints returned in the response.

Change-Id: I706a5b5be8002c633fe97b2429256735800a902e
Pass disk usage options when using --all

This commit allows the arguments --human-readable and --top to be
used with --all to customize the output of the disk usage portion.

There are no unit tests attached to this commit, I have created and
referenced bug #1344200 which shows there is a larger bug to address
to create several missing tests.

Change-Id: I46b8359533989efc3067971acaafec1d0cbc2b9f
Closes-Bug: #1344178
Related-Bug: #1344200
Add X-Delete-At/After support to FormPost

This change adds the ability to specify a X-Delete-At or
X-Delete-After attribute when using the FormPost middleware.

Here is an example of what you need to add to the form:

  <input type="hidden" name="x_delete_at" value="<unix-timestamp>"/>
  <input type="hidden" name="x_delete_after" value="<seconds>"/>

To be inline with the other form imput names, x-delete-at/after has
changed to x_delete_at/after.

DocImpact
Change-Id: Ib1cc0bcf1dd7d2b689f2f26d100f9bab36880c81
Closes-Bug: #1065522
lock_file race fixes

I attempted to use this function and found a few problems.

We shouldn’t unlink the file after closing it, because someone else could lock
it in between.  Switch to unlink before close.

If someone else locked the file between our open and flock, they are likely to
unlink it out from underneath us.  Then we have a lock on a file that no longer
exists.  So stat the filename after locking to make sure the inode hasn't
changed or gone away.

We probably shouldn’t unlink the file if we time out waiting for a lock.  So
move that to before the finally block.

Change-Id: Id1858c97805d3ab81c584eaee8ce0d43d34a8089
Fix a couple small things in formpost doc string

Change-Id: Idb8d4707180396662493aacbd1f125c4ca43a1ca
Merge "Add X-Delete-At/After support to FormPost"
typo tradeoff -> trade-off

Change-Id: Iad5eb611bc727991606f8a2d4dbb08dcc8fa3ddb
Merge "Add v2 API to list endpoints middleware"
Merge "typo tradeoff -> trade-off"
Merge "lock_file race fixes"
Remove duplicate conditional statement

It contains two same conditional statements to check whether the first
argurment is end with '.ring.gz'. The latter one can be covered by 'else'
clause.

Change-Id: I4b67dc330e82e793020bec4b74047062313ae11c
Signed-off-by: Lin Yang <lin.a.yang@intel.com>
Use auth_token from keystonemiddleware

auth_token middleware in python-keystoneclient is deprecated and has
been moved to the keystonemiddleware repo.

Change-Id: Ia04aa83348e0776cb3239cb5420ee1450a990d5b
Closes-Bug: #1342274
Fix trivial typos

Fixes a few typos I have stumbled across recently.

Change-Id: Ib232924f6b23c08578c52a8dd63aaaa8789f9da7
Merge "Fix a couple small things in formpost doc string"
Merge "Fix trivial typos"
Merge "Remove duplicate conditional statement"
Reuse a cursor instead of a conn.execute for better performance

In some limited testing it was ~10% faster. Every little bit helps right?

Change-Id: Ide55b7600d625ad151daca73fbdd8684f0f9e7f8
Merge "Reuse a cursor instead of a conn.execute for better performance"
Reuse a cursor instead of a conn.execute for better performance

Basically the same thing as https://review.openstack.org/#/c/109694/
but over in the account backend.

Change-Id: I9255f3145818da3adafc16d3bcf159608a78fe51
Fix "Handoff requested (N)" log spam

A long, long time ago, on a GET request, the proxy would go look on 3*
nodes for the requested thing. If one of the primary nodes was
error-limited, it'd look on two primaries and a handoff. Since this
indicated some failure somewhere, the proxy would emit a warning:
"Handoff requested (1)". If two primaries were down, there'd be a
second message "Handoff requested (2)", and so on.

Some StatsD messages were emitted too.

A somewhat shorter time ago (commit d79a67eb), the proxy started
looking into handoffs if it got 404s from the primaries. While this
was a good idea, it resulted lots of "Handoff requested (N)" log spam;
you'd see these messages on every single 404. Also, the StatsD
handoff_count and handoff_all_count metrics shot way up and turned
into noise.

This commit restores the original intent (and usefulness) of the log
messages and StatsD metrics: if the proxy only looks at the normal
number of handoff nodes, nothing is logged. However, if a primary is
down, then the message "Handoff requested (1)" will be logged,
indicating that the proxy looked at one more handoff than it normally
would, and this happened because a primary node was error-limited.

Closes-Bug: 1297214

* or whatever the replica count was

Change-Id: If1b77c18c880b096e8ab1df3008db40ce313835d