Do not use pickle for serialization in memcache, but JSON

We don't want to use pickle as it can execute arbitrary code. JSON is
safer. However, note that it supports serialization for only some
specific subset of object types; this should be enough for what we need,
though.

To avoid issues on upgrades (unability to read pickled values, and cache
poisoning for old servers not understanding JSON), we add a
memcache_serialization_support configuration option, with the following
values:

 0 = older, insecure pickle serialization
 1 = json serialization but pickles can still be read (still insecure)
 2 = json serialization only (secure and the default)

To avoid an instant full cache flush, existing installations should
upgrade with 0, then set to 1 and reload, then after some time (24
hours) set to 2 and reload. Support for 0 and 1 will be removed in
future versions.

Part of bug 1006414.

Change-Id: Id7d6d547b103b4f23ebf5be98b88f09ec6027ce4
Fix when rate_limit_after_segment kicks in.

If rate_limit_after_segment was 10 in the proxy config, then after 10
segments were coughed up by _load_next_segment() with no calls to
sleep(), the 11th segment would not trigger a sleep() call.  The 12th
segment triggered a sleep(0) call, but it was only after the 13th
segment was loaded that an actual rate-limiting (non-zero) sleep got
called.

With this patch, a rate_limit_after_segment of 10 will start sleeping
the correct amount after the 11th segment.

Updated proxy-server.conf-sample with rate_limit_after_segment and
rate_limit_segments_per_sec.

Change-Id: I937c366996e6d6ab47c614d6db470e3be9657c07
Remove redefinition of import traceback

Removed the redefinition of import in swift/object/server.py
Add my name in AUTHORS

Change-Id: Ia74df662de08fc52b92b828978d96a22b5df5597
Merge "Remove redefinition of import traceback"
Misc. swift-bench improvements.

swift-bench now honors the environment variables, ST_AUTH, ST_USER, and
ST_KEY like python-swiftclient does.

Added --lower-object-size (or -l) command-line option which, if
specified, will turn a specified --object-size into --upper-object-size.
It will raise a ValueError if --object-size is not specified or is <=
--lower-object-size.

BenchController how handles SIGINT (KeyboardInterrupt) similarly to the
swift command-line client: the first Ctrl-C will make it fast-track to
completion (no new PUT or GET operations are started, but everything PUT
is DELETE'ed).  A second Ctrl-C will immediately exit.  The behavior for
SIGTERM is unchanged (a single SIGTERM will immediately terminate the
process).

Added a sample configuration file for swift-bench, with documenting
comments.

Change-Id: I6f394ad995300fc8af3d565d95c3b45559ee510a
x-newest cleanup code with test. Fixes bug 1037337

Change-Id: Ie99250250171246e8c13e8d8c8258101bd78cce4
Remove the gettext wrapper of server_type.

The gettext wrapper of server_type in controllers is used to translate
server_type in log. But the transfer_headers function which used to
filter the illegal metadata headers use the wrapped server_type. So, it
will cause problems that 'X-Account-Meta-' and 'X-Container-Meta-' would
be filtered when the language of operating system is not english, such
as Chinese or Japnese. So, I removed the wrapper.

Change-Id: I7d47e67b4caf5d0be171a90740f9bb12fa2cba84
Fixes: Bug #1042142
fallocate call error handling

fallocate() failures properly return HTTPInsufficientStorage from
object-server before reading from wsgi.input, allowing the proxy
server to error_limit that node.

Change-Id: Idfc293bbab2cff1e508edf58045108ca1ef5cec1
Merge "x-newest cleanup code with test. Fixes bug 1037337"
Merge "Do not use pickle for serialization in memcache, but JSON"
Merge "Misc. swift-bench improvements."
Merge "Remove the gettext wrapper of server_type."
Added ability to disable fallocate

Change-Id: Id8872c581ed23378a8e14cbf3bf049b5c0d21577
Upating proxy-server StatsD logging.

Removed many StatsD logging calls in proxy-server and added
swift-informant-style catch-all logging in the proxy-logger middleware.
Many errors previously rolled into the "proxy-server.<type>.errors"
counter will now appear broken down by response code and with timing
data at: "proxy-server.<type>.<verb>.<status>.timing".  Also, bytes
transferred (sum of in + out) will be at:
"proxy-server.<type>.<verb>.<status>.xfer".  The proxy-logging
middleware can get its StatsD config from standard vars in [DEFAULT] or
from access_log_statsd_* config vars in its config section.

Similarly to Swift Informant, request methods ("verbs") are filtered
using the new proxy-logging config var, "log_statsd_valid_http_methods"
which defaults to GET, HEAD, POST, PUT, DELETE, and COPY.  Requests with
methods not in this list use "BAD_METHOD" for <verb> in the metric name.
To avoid user error, access_log_statsd_valid_http_methods is also
accepted.

Previously, proxy-server metrics used "Account", "Container", and
"Object" for the <type>, but these are now all lowercase.

Updated the admin guide's StatsD docs to reflect the above changes and
also include the "proxy-server.<type>.handoff_count" and
"proxy-server.<type>.handoff_all_count" metrics.

The proxy server now saves off the original req.method and proxy_logging
will use this if it can (both for request logging and as the "<verb>" in
the statsd timing metric).  This fixes bug 1025433.

Removed some stale access_log_* related code in proxy/server.py.  Also
removed the BaseApplication/Application distinction as it's no longer
necessary.

Fixed up the sample config files a bit (logging lines, mostly).

Fixed typo in SAIO development guide.

Got proxy_logging.py test coverage to 100%.

Fixed proxy_logging.py for PEP8 v1.3.2.

Enhanced test.unit.FakeLogger to track more calls to enable testing
StatsD metric calls.

Change-Id: I45d94cb76450be96d66fcfab56359bdfdc3a2576
Can run swift-bench across multiple cores/servers.

You run one or more swift-bench-client processes like this:
    $ swift-bench-client 127.0.0.1 20001
    $ swift-bench-client 127.0.0.1 20002

Then you run swift-bench with a new option, --bench-clients (-b), which is
specified once for each swift-bench-client:
    $ swift-bench -b 127.0.0.1:20001 -b 127.0.0.1:20002

You get log lines from each client (interleaved) along with a final report
for all clients:
    127.0.0.1:20002 swift-bench-server 2012-08-25 22:44:06,148 INFO Auth version: 1.0
    127.0.0.1:20001 swift-bench-server 2012-08-25 22:44:06,148 INFO Auth version: 1.0
    127.0.0.1:20001 swift-bench-server 2012-08-25 22:44:12,249 INFO 83 PUTS [0 failures], 41.5/s
    127.0.0.1:20002 swift-bench-server 2012-08-25 22:44:14,430 INFO 74 PUTS [0 failures], 34.3/s
    ...
    127.0.0.1:20002 swift-bench-server 2012-08-25 22:45:18,942 INFO Auth version: 1.0
    127.0.0.1:20002 swift-bench-server 2012-08-25 22:45:20,946 INFO 238 DEL [2 failures], 118.9/s
    swift-bench 2012-08-25 22:45:27,549 INFO 2000 PUTS **FINAL** [0 failures], 56.8/s
    swift-bench 2012-08-25 22:45:27,550 INFO 30000 GETS **FINAL** [50 failures], 974.6/s
    swift-bench 2012-08-25 22:45:27,550 INFO 2000 DEL **FINAL** [20 failures], 237.1/s

The concurrency, PUT count, and GET count config settings are divided by
the number of bench_clients.  In other words, the same volume of work is
attempted (vs. not specifying --bench-clients), but it can now span
servers and CPU cores.

Benchmark containers are created (if use_proxy = yes) and deleted (if
delete = yes), with appropriate concurrency, in the initiating
swift-bench process, not any of the swift-bench-client processes.

Change-Id: Idbf31a23093244ab357a9bf77e6031257774f24a
Fix PEP8 issues in ./test/unit/common .

Fix some pep8 issues in
       modified:   test_bufferedhttp.py
       modified:   test_constraints.py
       modified:   test_db.py
       modified:   test_db_replicator.py
       modified:   test_init.py
make the code looks pretty.

Change-Id: I1c374b1ccd4f028c4e4b2e8194a6d1c201d50571
bumped version to 1.7.0 to reflect current dev effort

Change-Id: I5423c8bf11b14f862cb321583d91991177fce5eb
use simplejson to serialize acct/cont results

I did some benchmarking, and it's just a lot faster to use simplejson
to serialize account and container listings.

Change-Id: Ib0f751d9a5b6f4020031e952986811f0be090cce
Fall back to UDP if /dev/log does not exist.

If /dev/log does not exist, swift.common.utils.get_logger would fail as
it only handles the situation where /dev/log isn't a socket.

Fixes bug 1045820

Change-Id: I2c33e0fa7f88f03d5b0495779832ed4de7c496da
Update swift.common.db to us swift.cmmn.utils.json

Change-Id: Ib4baac7b8b24d7fd982c71591deda3a096f470f2