Updated probe tests

The probe tests were woefully out of date with all the changes that
have ocurred since they were written. I've updated most of them and
removed some that are hopeless outdated.

I also greatly improved the timing issues (hopefully completely
solved them? I ran them 25 times with no problems) and made them pep8
1.3.1 safe.

Change-Id: I8e9dbb6e7d6e04e293843b1dce1ded99d84e0348
Report all unmounted drives

- swift-recon cli now reports all unmounted drives.
- Fixes bug 1031060

Change-Id: Ia9e6bbf05215879b602e9dcb77d007094c494dc1
Ensure parameters sent to db are utf8 strs

Change-Id: I45952adfcfbc813477ee9966f05dcdeef23c683b
Do not use pickle for serialization in memcache, but JSON

We don't want to use pickle as it can execute arbitrary code. JSON is
safer. However, note that it supports serialization for only some
specific subset of object types; this should be enough for what we need,
though.

To avoid issues on upgrades (unability to read pickled values, and cache
poisoning for old servers not understanding JSON), we add a
memcache_serialization_support configuration option, with the following
values:

 0 = older, insecure pickle serialization
 1 = json serialization but pickles can still be read (still insecure)
 2 = json serialization only (secure and the default)

To avoid an instant full cache flush, existing installations should
upgrade with 0, then set to 1 and reload, then after some time (24
hours) set to 2 and reload. Support for 0 and 1 will be removed in
future versions.

Part of bug 1006414.

Change-Id: Id7d6d547b103b4f23ebf5be98b88f09ec6027ce4
To simulate the real workloads, the objects to be uploaded could be created in the random sizes, which are bounded (lower_object_size and upper_object_size) by the user inputs.

Change-Id: I64647c5d613a0794f0807886a4833cd5c31c0c5e
Use custom encoding for RingData, not pickle.

Serialize RingData in a versioned, custom format which is a combination
of a JSON-encoded header and .tostring() dumps of the
replica2part2dev_id arrays.  This format deserializes hundreds of times
faster than rings serialized with Python 2.7's pickle (a significant
performance regression for ring loading between Python 2.6 and Python
2.7).  Fixes bug 1031954.

swift.common.ring.ring.RingData is now responsible for serialization and
deserialization of its data via a new load() class method and save()
object method.  The new implementation is backward-compatible; if a ring
does not begin with a new-style magic string, it is assumed to be an
old-style pickle-dumped ring and is handled as before.  So new Swift
code can read old rings, but old Swift code will not be able to read
newly-serialized rings. THIS SHOULD BE MENTIONED PROMINENTLY IN THE
RELEASE NOTES.

I didn't want to bite of more than necessary, so I didn't mess with
builder file serialization.

Change-Id: I799b9a4c894d54fb16592443904ac055b2638e2d
all in one setup rc.local needs -p for mkdir command

bug fixes for swift 1033089
Swift 1.6.0 All in One setup uses rc.local to create a set of
directories like the following:
   /var/cache/swift /var/cache/swift2 /var/cache/swift3
   /var/cache/swift4
   /var/run/swift
without -p for mkdir command, when some of these directories
already exist, the rc.local fails so that the next set of
command won't execute. This causes swift fail to start because
of /var/run/swift does not exist. Adding -p to both mkdir
command solves this problem.

Change-Id: Ia26c9a857723747008293c0631c9acc1b58eca7e
Merge "Report all unmounted drives"
Merge "all in one setup rc.local needs -p for mkdir command"
Fixed bug where expirer would get confused by...

Fixed bug where expirer would get confused by odd deletion times.
Since this has already rolled out, I just capped things at ten 9s, or
Sat Nov 20 17:46:39 2286. I can't wait for the Y2286 world panic. :/

Change-Id: Iba10963faa344a418a1fa573d5c85f4ff864b574
update object versions docs

Change-Id: I00bf2e20c2683d0440ca425461e0ab87197ee720
handle exception correctly in _make_app_iter_reader

_make_app_iter_reader uses a queue to pass read data to
_make_app_iter, and uses an empty string as a terminator.  The problem
is that it pushes the same terminator even if it fails to reads the
data.  If an exception happens before pushing all data to the queue,
_make_app_iter exits without yielding all data.  In that case,
proxy-server stops in the middle of sending data, and a client waits
for all data to come infinitely.

This patch uses a boolean value for the terminator instead of an empty
string to distinguish the results, and raises an exception in the
error cases.  Fixes bug 890888.

Change-Id: Ie4d48025843aa1bb600446da6f4fc49f82a440bf
Merge "Use custom encoding for RingData, not pickle."
ring.py: Uses simplejson if avail and pep8 1.3.1

Change-Id: I1f91a82da064c9f0e64779f4fa8951962678831b
Cleanup after LockTimeout

- If lock_file encounters a LockTimeout a fd can be left behind
- Fixes bug 1038191

Change-Id: Ib3f45d18aeeec91eedf0b4f0492664d33b4f57fa
Merge "update object versions docs"
Merge "To simulate the real workloads, the objects to be uploaded could be created in the random sizes, which are bounded (lower_object_size and upper_object_size) by the user inputs."
Merge "handle exception correctly in _make_app_iter_reader"
A few misc. code cleanups.

Fixed a couple config gets which weren't using TRUE_VALUES (hopefully
the last ones?).

Removed a deprecated parameter from
swift.common.db.ContainerBroker.list_objects_iter() a "TODO" which dates
back to the original import of Swift into github.

Also made AccountBroker.reclaim()'s docstring match its arg names.

Change-Id: I646bdd00a3693763b55a50ba210c7ed2ba1df4cd
Retuen a reasonable response to client.

As we use the 'account_autocreate' for auto-create accounts, we send a
request to HEAD an account which is just deleted(the account-reaper not
reclaim it immediately), the proxy-server will return 'HTTP 500 Internal
Server Error'.
In my opinion, this is unreasonable, so I change the code in swift/proxy
/server.py for returning a reasonable response. I modified the code in
POST, GETorHEAD and account_info. At last, I modified some code in the
unittest test/unit/proxy/test_server.py.

Change-Id: Ib057b387c9da073d707ffae49ead54206a8fb7dd
Fixes: Bug #1037889