Checking in changes prior to tagging of version 2.68.

Changelog diff is:

diff --git a/CHANGES b/CHANGES
index f0f578c..b74f7f4 100644
--- a/CHANGES
+++ b/CHANGES
@@ -1,3 +1,35 @@
+2013-08-07: Release version 2.68
+
+   * optimize monitor worker for large installs (Eric Wong <normalperson@yhbt.net>)
+
+   * allow startup without job_master (and dependent workers) (Eric Wong <normalperson@yhbt.net>)
+
+   * store: do not disconnect for max_handles while locked (Eric Wong <normalperson@yhbt.net>)
+
+   * store: do not auto-reconnect while holding a lock (Eric Wong <normalperson@yhbt.net>)
+
+   * add naive MultipleDevice replication policy (Eric Wong <normalperson@yhbt.net>)
+
+   * httpfile: log mogstored I/O errors when checksumming (Eric Wong <normalperson@yhbt.net>)
+
+   * ProcManager: log socketpair errors correctly (Eric Wong <normalperson@yhbt.net>)
+
+   * fix "drain" handling used by MultipleHosts replpolicy (Eric Wong <normalperson@yhbt.net>)
+
+   * httpfile: correct timeouts for sidechannel digest (Eric Wong <normalperson@yhbt.net>)
+
+   * httpfile: correct FILE_MISSING check in digest_mgmt (Eric Wong <normalperson@yhbt.net>)
+
+   * mogstored: avoid bareword on IO::AIO w/o fadvise (Eric Wong <normalperson@yhbt.net>)
+
+   * ProcManager: only log times_out_of_qworkers for new queries (Eric Wong <normalperson@yhbt.net>)
+
+   * Don't emit warnings if we're lacking the space free of a device.  If we don't find space on devices with known space free, try the unknowns. (Dave Lambley <davel@state51.co.uk>)
+
+   * list_keys: escape in Store, allow [%\\] as prefix (Eric Wong <normalperson@yhbt.net>)
+
+   * list_keys: consistent ESCAPE usage across DB types (Eric Wong <normalperson@yhbt.net>)
+
 2013-02-02: Release version 2.67

    * Serialize tempfile reaping (dormando <dormando@rydia.net>)
move Danga::Socket->Reset to ProcManager

We will be using Danga::Socket in more (possibly all) workers,
not just the Monitor and Reaper.  Resetting in workers that do
not use Danga::Socket is harmless and will not allocate
epoll/kqueue descriptors until the worker actually uses
Danga::Socket.
monitor: refactor/rewrite to use new async API

In order to migrate to the upcoming Danga::Socket-based
HTTP API, we'll first refactor monitor to use the new API
(but preserve LWP usage behind-the-scenes).

DEBUG=1 users will see the elapsed time for all device refreshes
each time monitor runs.

While we're at it, also guard against race conditions on the
PUT/GET test by double-checking on failure.  (A long-standing
TODO item)

also squashed the following commit:

  use conn_timeout in monitor, node_timeout in other workers

  This matches the behavior in MogileFS:Server 2.65.

  It makes sense to use a different, lower timeout in monitor to
  quickly detect overloaded nodes and avoid propagating their
  liveness for a monitoring period.

  It also makes sense to use a higher value for node_timeout in
  other workers since other actions are less fault-tolerant.

  For example, a timed-out size check in create_close may cause a
  client to eventually reupload the file, creating even more load
  on the cluster.
monitor: switch to non-blocking HTTP device checks

Net::HTTP::NB is usable with Danga::Socket and may be used to
make HTTP requests in parallel.

The new connection pool supports persistent connection pooling
similar to LWP::ConnCache.  Total connection capacity is
enforced to prevent out-of-FD situations on the workers.

Unlike LWP::ConnCache, MogileFS::ConnectionPool is designed for
use with concurrent, active connections.  It also supports
queueing (when any enforced capacity or system limits are
reached) and relies on Danga::Socket for scheduling queued
connections.

In addition to total capacity limits, MogileFS::ConnectionPool
also supports limiting concurrency on a per-destination basis to
avoid potentially overloading a single destination.

Currently, we limit ourselves to 20 connections from a single
worker (matching the old LWP limit) and also limit ourselves
to 20 connections to a single host (again matching our previous
LWP behavior).
JobMaster: use Danga::Socket to schedule

In the future, this will allow JobMaster to write concurrently to
ProcManager (or even individual workers) without blocking.

(tweaked to accomodate "!want 0 job_master" support)
httpfile: remove size check failure backoff handling

This backoff handling in HTTPFile is redundant for several reasons:

* We rely on the monitor worker anyways to inform us of unreachable hosts

* Monitor runs much faster nowadays, giving us a smaller window for
  out-of-date information about host reachability

* HTTPFile->size no longer connects to the sidechannel port,
  only HTTP, so we waste fewer syscalls on failure if we a
  host went down before the last monitor run.
fsck: parallelize size checks for any given FID

This allows us to us to speed up fsck on high latency clusters
by issuing parallel HEAD requests.
httpfile: use Net::HTTP::NB, remove LWP::UserAgent

This allows us to use the same HTTP connections between
digest and HTTP size checks, reducing the number of open
connections we need in the Fsck worker.
httpfile: use HTTP connection pool for DELETE

This simplifies the delete subroutine and should reduce
the number of sockets created during rebalance.
delete worker uses persistent HTTP connections

This allows us to avoid running ourselves out of local ports
when handling massive delete storms.

Eventually, we can parallelize deletes in a manner similar
to fsck size checking.
device: reuse HTTP connections for MKCOL

This can reduce latency for folks still stuck with MKCOL.
This creates no new sockets for replicate and monitor in
all cases, as connections to the HTTP DAV server are already
used in those workers.

This only adds new persistent connections to the queryworker if
GET-only HTTP ports are configured (queryworker already may call
HTTPFile->size).
create_open: parallelize directory vivification

For setups stuck needing MKCOL, we can parallelize
directory vivification for multi-destination uploads.
replicate: enforce expected Content-Length in http_copy

There's no reason we should ever skip Content-Length validation
if we know which FID we're replicating and have an FID object
handy.

Conflicts:
	lib/MogileFS/Worker/Replicate.pm
replicate: use persistent connection from pool if possible

This should reduce the amount of TIME-WAIT sockets and TCP
handshakes when replicating, especially with small files.

An attempt was previously made to use the Net::HTTP::NB API
directly, but that resulted in complicated callback nesting
and state management needed to throttle the reader if the
sender socket were blocked in any way.

There were many bugs in the early version of this code as
a result of the complicated code.  Even after all the bugs
got fixed, a small performance reduction due to the extra
buffer copies was difficult to avoid.

Thus I started using the synchronous version to keep the code
simple and fast while preserving the ability to use persistent
sockets to avoid excessive TIME-WAIT and handshaking for small
file replication.
host: handle case where conn_get may return undef

MogileFS::ConnectionPool::conn_get may return undef on some
errors, so we must account for that and not kill the replicate
worker.
ConnectionPool: improve reporting of socket creation errors

Send the entire error message (including intended host:port so
it is more informative when it propagates to
Connection::HTTP::err_response.  We also do not need to log
the error in ConnectionPool, as the error will be logged by
the caller.

While we're at it, fix the documentation and a spelling error in
err_response, too.
t/http.t: test error handling on non-running server

We need to ensure we don't blow up a worker process if a
server is shutdown and a connection attempted before the
monitor notices.
connection/{poolable,http}: common retry logic for timeouts

We will want similar logic for Mogstored sidechannel to avoid
retrying on timeout.
connection/poolable: stricter timeout key check

String representations of small floating point values may
be in (scientific) E notation, so we must ensure the entire
string is free of decimal digits before considering it a
configuration key.
ProcManager: SetAsChild drops inherited IPC sockets

Workers only need to inherit the minimum amount necessary from the
parent ProcManager.  Keeping the socket of unrelated workers in each
worker is wasteful and may contribute to premature resource
exhaustion.

Additionally, we will be using Danga::Socket in more (possibly all)
workers, not just the Monitor and Reaper.  Resetting in workers that
do not use Danga::Socket is harmless and will not allocate
epoll/kqueue descriptors until the worker actually uses
Danga::Socket.