Fix account quota mw for handling a bad source path

The copy source must be container/object.
This patch avoids the server to return
an internal server error when user provides
a path without a container.

Change-Id: I8ea4e62b2a00c4b4a1666d30411a6c93a4f848a5
Fixes: bug #1254626
Update tox.ini so flake8 works even without hacking

The flake8 command (when run from the command line or via vim-flake8) seems to
bypass the the normal pep8 [W]arning and [E]rror as well as py[F]lake checks
if the project level tox.ini includes a select, but doesn't include them in
it's select option (tested on flake8 v2.0 and v2.1).  Somehow if hacking is
installed these checks always run even if they're not explicitly listed in the
select list - but adding them so flake8 works on our project even without
hacking doesn't seem to cause any problems.

Also The [flake8] section's "select" option in tox.ini doesn't seem to run all
checks when presented with the space-comma seperated list.  I had to pull out
the bare except check (H201), because our hacking fix for the "H201 # noqa"
hasn't been released to pypi yet (maybe hacking version 0.9.0?).

As far as I know the Swift core development team is fully onboard with pep8,
and the flake8 checks are mostly like "unused import" stuff - so I *think*
this change is more about making the tool test what we want it too and not
changing or adding any new requirements or expectations.

Change-Id: Ia911c63745100cce182d5c5eb67b77e31c1e9b18
Fix ineffective expect header test

The test for expect header being sent
with non-zero object PUTs doesn't seem
to check its results, so passes whether
the expect header is set or not.

Change-Id: Ic74233c19e48aba43e63501467565e305ed399c3
Add missing swift constraint for functional tests

Looks like we are keeping at least four lists of the default
constraints in various places, and when we added the max_header_size
in initial Static Large Object Support commit (5d73da15) we failed to
add it to the two locations in the test tree (test/sample.conf and
test/functional/tests.py).

With this commit we take a small step to consolidate the list of
constraints to the constraints module by removing the list from the
tests module (note we still have them listed in the sample swift.conf
file and the test module's sample.conf file), document the missing
reference in the test module's sample.conf file, and, while we're at
it, use the SWIFT_CONF_FILE reference where possible.

Change-Id: Ic4fa5f1aedec9bc7a7b43ce1a2ca3cdebce6171f
Added a test for empty metadata

Here added a test for setting empty object metadata
and checking its value in response headers.

Change-Id: I460302661d150364a95bcd7f0ebbbc2a1e95507a
Functional tests for TempURL and SLO together

Change-Id: I4893911e50005d5b669b96e7d59b6149c734780e
Allow specification of object devices for audit

In object audit "once" mode we are allowing the user to specify
a sub-set of devices to audit using the "--devices" command-line
option.  The sub-set is specified as a comma-separated list.  This
patch is taken from a larger patch to enable parallel processing
in the object auditor.

We've had to modify recon so that it will work properly with this
change to "once" mode.    We've modified dump_recon_cache()
so that it will store nested dictionaries, in other words it will
store a recon cache entry such as {'key1': {'key2': {...}}}. When
the object auditor is run in "once" mode with "--devices" set the
object_auditor_stats_ALL and ZBF entries look like:
{'object_auditor_stats_ALL': {'disk1disk2..diskn': {...}}}. When
swift-recon is run, it hunts through the nested dicts to find the
appropriate entries.  The object auditor recon cache entries are set
to {} at the beginning of each audit cycle, and individual disk
entries are cleared from cache at the end of each disk's audit cycle.

DocImpact

Change-Id: Icc53dac0a8136f1b2f61d5e08baf7b4fd87c8123
Fix ring-builder crash

If you had a partition with 1 replica on a deleted device, 1 on a
zero-weight device, and 1 on a normal device, then rebalancing would
crash. This was because the ring builder was memoizing tiers_for_dev()
for all *available* devices (i.e. weight > 0 and not deleted), but
depended on it being present for all devices with partitions still on
them.

Since the builder moved the replica from the deleted device, it left
the one on the zero-weight device alone, so you had an unavailable
device with a partition replica still on it, triggering the crash.

Now we go ahead and memoize tiers_for_dev() for *all* devices, not
just available ones, thereby fixing the crash.

Change-Id: Ie0b58b65e0353732cf785ab772e95e699f3a5b5d
Make cors work better.

CORS doesn't really work with swift right now. OPTIONS calls for the most part
work but for so called "simple cross-site requests" (i.e. those that don't
require a pre-flight OPTIONS request) Swift always returns the Origin it was
given as the Access-Control-Allow-Origin in the response. This makes CORS
"work" for these requests but if you actually wanted the javascript user agent
to restrict anything for you it wouldn't be able to!

You can duplicate the issue with updated CORS test page:

http://docs.openstack.org/developer/swift/cors.html#test-cors-page

And a public container with an 'X-Container-Meta-Access-Control-Allow-Origin'
that does NOT match the webserver hosting the test-cors-page.

e.g.

with a public container that accepts cross-site requests from "example.com":

`swift post cors-container -m access-control-allow-origin:example.com -r .r:*`

You could point your browser at a copy of the test-cors-page on your
filesystem (the browser will will send 'Origin: null')

Without a token the XMLHttpRequest will not request any custom headers (i.e.
Access-Control-Request-Headers: x-auth-token) and the request will be made
with-out a preflight OPTIONS request (which Swift would have denied anyway
because the origin's don't match)

i.e. fill in "http://saio:8080/v1/AUTH_test/cors-container" for "URL" and
leave "Token" blank.

You would expect that the browser would not complete the request because
"Origin: null" does not match the configured "Access-Control-Allow-Origin:
example.com" on the container metadata, and indeed with this patch - it won't!

Also:

The way cors is set up does not play well with certain applications for swift.
If you are running a CDN on top of swift and you have the
Access-Control-Allow-Origin cors header set to * then you probably want the *
to be cached on the the CDN, not the Origin that happened to result in an
origin request.

Also:

If you were unfortunate enough to allow cors headers to be saved directly
onto objects then this allows them to supersede the headers coming from the
container.

NOTE: There is a change is behavior with this patch. Because its cors, a
      spec that was created only to cause annoyance to all, I'll write out
      what's being changed and hopefully someone will speak up if it breaks
      there stuff.

previous behavior: When a request was made with a Origin header set the
                   cors_validation decorator would always add that origin as
                   the Access-Control-Allow-Origin header in the response-
                   whether the passed origin was a match with the container's
                   X-Container-Meta-Access-Control-Allow-Origin or not, or even
                   if the container did not have CORS set up at all.
new behavior: If strict_cors_mode is set to True in the proxy-server.conf
              (which is the default) the cors_validation decorator will only
              add the Access-Control-Allow-Origin header to the response when
              the request's Origin matches the value set in
              X-Container-Meta-Access-Control-Allow-Origin. NOTE- if the
              container does not have CORS set up it won't just magically start
              working. Furthremore, if the Origin doesn't match the
              Access-Control-Allow-Origin - a successfully authorized request
              (either by token or public ACL) won't be *denied* - it just
              won't include the Access-Control-Allow-Origin header (it's up
              to the security model in the browser to cancel the request
              if the response doesn't include a matching Allow-Origin
              header).  On the other hand, if you want to restrict requests
              with CORS, you can actually do it now.

              If you are worried about breaking current functionality you
              must set:

              strict_cors_mode = False

              in the proxy-server.conf. This will continue with returning the
              passed in Origin as the Access-Control-Allow-Origin in the
              response.

previous: If you had X-Container-Meta-Access-Control-Allow-Origin set to *
          and you passed in Origin: http://hey.com you'd get
          Access-Control-Allow-Origin: http://hey.com back. This was true for
          both OPTIONS and regular reqs.
new: With X-Container-Meta-Access-Control-Allow-Origin set to * you get * back
     for both OPTIONS and regular reqs.

previous: cors headers saved directly onto objects (by allowing them to be
          saved via the allowed_headers config in the object-server conf)
          would be overridden by whatever container cors you have set up.
new: For regular (non-OPTIONS) calls the object headers will be kept. The
     container cors will only be applied to objects without the
     'Access-Control-Allow-Origin' and 'Access-Control-Expose-Headers' headers.
     This behavior doesn't make a whole lot of sense for OPTIONS calls so I
     left that as is. I don't think that allowing cors headers to be saved
     directly onto objects is a good idea and it should be discouraged.

DocImpact

Change-Id: I9b0219407e77c77a9bb1133cbcb179a4c681c4a8
Merge "Add missing swift constraint for functional tests"
Merge "Functional tests for TempURL and SLO together"
DRY out common.constraints a bit more

Change-Id: If8a71692b79e381036b34245e09697ac1009beca
make probetests work with conf.d configs

Change-Id: I451ff4629730a334ac1bd8fc6cd75de95314e153
Merge "Fix ring-builder crash"
Merge "DRY out common.constraints a bit more"
Merge "make probetests work with conf.d configs"
Merge "Make cors work better."
Merge "Added a test for empty metadata"
Allow pre-1970 dates in If-[Un]Modified-Since

If I want to fetch an object only if it is newer than the first moon
landing, I send a GET request with header:

    If-Modified-Since: Sun, 20 Jul 1969 20:18:00 UTC

Since that date is older than Swift, I expect a 2xx response. However,
I get a 412, which isn't even a valid thing to do for
If-Modified-Since; it should either be 2xx or 304. This is because of
two problems:

(a) Swift treats pre-1970 dates as invalid, and

(b) Swift returns 412 when a date is invalid instead of ignoring it.

This commit makes it so any time between datetime.datetime.min and
datetime.datetime.max is an acceptable value for If-Modified-Since and
If-Unmodified-Since. Dates outside that date range are treated as
invalid headers and thus are ignored, as RFC 2616 section 14.28
requires ("If the specified date is invalid, the header is ignored").

This only works for dates that the Python standard library can parse,
which on my machine is 01 Jan 1 to 31 Dec 9999. Eliminating those
restrictions would require implementing our own date parsing and
comparison, and that's almost certainly not worth it.

Change-Id: I4cb4903c4e5e3b6b3c9506c2cabbfbda62e82f35
Merge "Fix ineffective expect header test"