Handle DB creation race condition

Till now if a race occurs on a request to a DB when it is
first created, the loosing party die with 500...
... and an error stating a race is in the log file.

Both removed by simply accepting the fact that the DB is in fact
created by someone else - just less work for us to do... ;)

Fixes Bug #1051998

Change-Id: I8e29bf362edf2b1a84f9bffa39c04d5b3dc241b2
Rework to support RFC 2616 Sec 4.4 Message Length

RFC 2616 Sec 4.4 Message Length describes how the content-length and
transfer-encoding headers interact. Basically, if chunked transfer
encoding is used, the content-length header value is ignored and if
the content-length header is present, and the request is not using
chunked transfer-encoding, then the content-length must match the body
length.

The only Transfer-Coding value we support in the Transfer-Encoding
header (to date) is "chunked". RFC 2616 Sec 14.41 specifies that if
"multiple encodings have been applied to an entity, the
transfer-codings MUST be listed in the order in which they were
applied." Since we only supported "chunked". If the Transfer-Encoding
header value has multiple transfer-codings, we return a 501 (Not
Implemented) (see RFC 2616 Sec 3.6) without checking if chunked is the
last one specified. Finally, if transfer-encoding is anything but
"chunked", we return a 400 (Bad Request) to the client.

This patch adds a new method, message_length, to the swob request
object which will apply an algorithm based on RFC 2616 Sec 4.4
leveraging the existing content_length property.

In addition to these changes, the proxy server will now notice when
the message length specified by the content-length header is greater
than the configured object maximum size and fail the request with a
413, "Request Entity Too Large", before reading the entire body.

This work flows from https://review.openstack.org/27152.

Change-Id: I5d2a30b89092680dee9d946e1aafd017eaaef8c0
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Move parameter checking before disk accesses

For HEAD and GET requests on both containers and accounts APIs we now
perform parameter checking before we access the database. The unit
tests were updated to show that the parameter checking is performed
before accesses are performed.

Change-Id: Ieb753316cdccabf45022e3d83522d87d34aa6b0e
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Subtle change to tempurl content-disposition names

If an object name ends with a /, the content-disposition filename for
a tempurl ended up as an empty string. This fix just strips any
trailing slashes before calling basename.

Also, I added a test to verify that if the filename is passed as an
overriding query parameter, that it is used in full, no rstripping,
no basenaming.

Change-Id: I37725b6ded04ed3b91cdb21132490fd857276e2f
In TempURL, cache "no keys" for a shorter time.

In other places in the codebase, we cache a negative result for less
time (usually a tenth) than we cache a positive result. This commit
makes TempURL do the same.

This creates a slightly nicer experience for someone trying to set up
TempURL; if they get the initial key setup wrong, try it, then fix it,
it now takes only 6 seconds until they can use signed URLs, not
60. That's a short enough time that they may not even notice the
caching of no keys.

Change-Id: I521f023e7cddaecd07f0ce32aedd4059bd0b8ec4
Merge "Subtle change to tempurl content-disposition names"
Merge "Move parameter checking before disk accesses"
Merge "Handle DB creation race condition"
Rename requires files to standard names.

Rename tools/pip-requires to requirements.txt and tools/test-requires
to test-requirements.txt. These are standard files, and tools in the
general world are growing intelligence about them.

Change-Id: Ib3e50a811868e2969923d978ee00c4f92682aa1c
Fixes: bug #1179008
Mock SysLogHandler for proxy/test_server.py

Our tests get rather grabby with file descriptors, mostly due to the
SysLogHandler. In recent work I'm doing, my additional tests put
things just over the 1024 limit and tests would start to fail
somewhat randomly. This alleviates the problem by mocking out one of
the bigger users of SysLogHandler, proxy/test_server.py. If you want
to verify this has an impact, you can put the following in your
proxy/test_server.py's overall teardown method and try it with and
without the mocking:

    import subprocess
    print >>sys.stderr, '%s OPEN FILES' % len(subprocess.Popen(
        ['lsof'], stdout=subprocess.PIPE).communicate()[0].split('\n'))

Change-Id: I1bd3efe46ee69d09a32c5a964f04e36e46506446
Fix faked-out account GET for JSON and XML.

If account autocreation is on and the proxy receives a GET request for
a nonexistent account, it'll fake up a response that makes it look as
if the account exists, but without reifying that account into sqlite
DB files.

That faked-out response was just fine as long as you wanted a
text/plain response, but it didn't handle the case of format=json or
format=xml; in those cases, the response would still be
text/plain. This can break clients, and certainly causes crashes in
swift3. Now, those responses match as closely as possible.

The code for generating an account-listing response has been pulled
into (the new) swift.account.utils module, and both the fake response
and the real response use it, thus ensuring that they can't
accidentally diverge. There's also a new probe test for that
non-divergence.

Also, cleaned up a redundant matching of the Accept header in the code
for generating the account listing.

Note that some of the added tests here pass with or without this code
change; they were added because the code I was changing (parts of the
real account GET) wasn't covered by tests.

Bug 1183169

Change-Id: I2a3b8e5d9053e4d0280a320f31efa7c90c94bb06
Merge "Rework to support RFC 2616 Sec 4.4 Message Length"
Replaced <exc>.message with str(<exc>)

Change-Id: I9a030b84dafe2b2300e8735052f6f15f34bc7aa7
Merge "Mock SysLogHandler for proxy/test_server.py"
implement an rsync_bwlimit setting for object replicator

Change-Id: I8789d6e4d22de83db9a2760d51a94eb56a48c3b5
Merge "Rename requires files to standard names."
python3: Introduce py33 to tox.ini

Introduce py33 to tox.ini to make testing with
python3 easier.

Change-Id: Id5c76d58551af4671f30ace85bdbbf5803a58368
Signed-off-by: Chuck Short <chuck.short@canonical.com>
Add requirements files to the source distribution

Also remove the tools directory, since it is no longer used, now that
https://review.openstack.org/30887 is merged, which renamed the
requirements files to requirements.txt and test-requirements.txt.

Fixes bug 1186625

Change-Id: Icf1ef1d2d83140420163c25257d4802282063cfa
Merge "Fix faked-out account GET for JSON and XML."
Change to use dash instead of slash

There is no known way to embed slashes into container names, even when
using URL escape sequences. Since it would seem that the more common
case would be some other character, we validate using a dash ('-')
character as the delimiter.

It should also be noted that the documentation for container listing,
found here:
http://docs.openstack.org/api/openstack-object-storage/1.0/content/s_listcontainers.html,
does not describe the use of the prefix or delimiter parameters.

DocImpact

Change-Id: Ie467a4181c6f64b24c82c72b70a37183d443dee1
Signed-off-by: Peter Portante <peter.portante@redhat.com>