When a mogstored is down, die with a more informative message.
iostat: allow MOG_IOSTAT_CMD env override

This makes it easier to test mock or alternative iostat
implementations.  This can be used for emulating the
iostat output on other platforms.
iostat: increase flexibility of iostat parser

The parser now looks for contiguous lines of statistics and (if
it's previously captured stats) emits whenever the first
non-stats line appears.  Relying on the "Device:" line is not
portable to FreeBSD (and possibly other iostats implementations).
The parser also ignores leading/trailing whitespace on each
statistics line.

Tested on Linux (sysstat 10.0.5) and FreeBSD 9.

For testing iostat output on FreeBSD, I used MOG_IOSTAT_CMD
like this on my GNU/Linux system:

  MOG_IOSTAT_CMD="ssh fbsd9vm iostat -dx 1 30" mogstored ...

ref: http://code.google.com/p/mogilefs/issues/detail?id=9
remove old rebalance knobs from server settings

This hasn't been used since the old rebalance code was nuked
for 2.40 (commit 0be2f9771a213e92d807f29bead13e6e4203ad54)
fix tests when /etc/mogilefs/mogstored.conf exists

mogstored gains a --skipconfig switch which we use in tests to
ignore the default config file.  mogilefsd has had this switch
(with identical semantics) since 2004.
tests: add basic test for reaper

Reaper isn't tested anywhere else.  We plan on changing it
slightly so ensure we don't introduce regressions.
reaper: factor out reap_fid sub from the work loop

A subroutine with a 20-line comment deserves to be its own sub.
This will make it easier to see what the future reaper lock
will guard without needing to scroll on small terminals.
reaper: global lock around DB interaction

We don't want multiple reaper process stepping on each other
during UPDATE/INSERT, causing needless conflicts/failures at the
DB level for every single FID.

JobMaster already locks its queues in a similar way to prevent
conflicts, so this should not noticeably harm performance (and
may improve performance due to the DB conflict reduction).
reaper: add "queue_rate_for_reaper" server setting

This controls the number of FIDs the reaper can inject into the
replication queue for each dead device, per wakeup.

This defaults to 1000, the same value its had since (at least) 2006.
move ENDOFTIME constant from replicate to store

This will make it easier to reuse this constant in other
workers that can check the queues (e.g. job_master/reaper).
reaper: add queue_size_for_reaper server setting

Users may now configure the queue_size_for_reaper server setting to
limit the size of the non-urgent replication queue.

The urgent replication queue (nexttry == 0) is unaffected,
as are other processes (fsck) which may inject into the
replication queue.

The default remains unlimited, the reaper will queue as fast as
it possibly can: 1000 FIDs every 5 seconds (per process)
reaper: switch to Danga::Socket for scheduling

Reaper now schedules the first batch of files on a newly-dead
device for replication after delaying (on the reaper itself) for
DEVICE_SUMMARY_CACHE_TIMEOUT+1 (16) seconds.  This allows all
subsequent files to replicate sooner, without the 16s delay.
Since Danga::Socket is used to schedule this 16s delay, reaping
of other dead devices won't be impacted by this delay.

The DEVICE_SUMMARY_CACHE_TIMEOUT+1 delay still remains to
offer a small level of protection against replicators with
out-of-date internal device caches and writable-but-"dead"
devices.

As an additional countermeasure against out-of-date device
caches, reapers will now slowly back off of a device over the
course of 4 hours after it fails to find new, unreaped FIDs.
Previously, any files that got replicated to an already "dead"
device would remain there until a reaper restart.
reaper: remove update_devcount call

Since reaper now schedule replications with the same priority as
fsck, we will also rely on the replicator to call
update_devcount for us, allowing us to avoid making an
unnecessary write to the database.
reaper: better handling of lock failures

The delay backoff should only occur if we got a successful lock.
Backing off the delay on lock failure can result in the
delay becoming undef and (incorrectly) making the reaper give up
on a device.

Fortunately, lock failures with the extremely long (60s) lock
timeout is unlikely to be a problem in practice.
Postgres advisory lock instead of table-based lock

Update Pg locking code to use Postgres advisory locks. Now requires
Postgres 8.4 as min version.

Signed-off-by: Robin H. Johnson <robbat2@gentoo.org>
Cleanup lock timeout sleep location per Eric.

Signed-off-by: Robin H. Johnson <robbat2@gentoo.org>
Checking in changes prior to tagging of version 2.65.

Changelog diff is:

diff --git a/CHANGES b/CHANGES
index c552089..64455f4 100644
--- a/CHANGES
+++ b/CHANGES
@@ -1,3 +1,28 @@
+2012-08-13: Release version 2.65
+
+   * Postgres advisory lock instead of table-based lock (Robin H. Johnson <robbat2@gentoo.org>)
+     Now requires minimum Postgres version of pg8.4.
+
+   * reaper: switch to Danga::Socket for scheduling (Eric Wong <normalperson@yhbt.net>)
+
+   * reaper: add queue_size_for_reaper server setting (Eric Wong <normalperson@yhbt.net>)
+
+   * reaper: add "queue_rate_for_reaper" server setting (Eric Wong <normalperson@yhbt.net>)
+     defaults to 1000, same as previously.
+
+   * reaper: global lock around DB interaction (Eric Wong <normalperson@yhbt.net>)
+     prevents reapers clobbering each other, causing a reduction in DB writes.
+
+   * tests: add basic test for reaper (Eric Wong <normalperson@yhbt.net>)
+
+   * fix tests when /etc/mogilefs/mogstored.conf exists (Eric Wong <normalperson@yhbt.net>)
+
+   * iostat: increase flexibility of iostat parser (Eric Wong <normalperson@yhbt.net>)
+
+   * iostat: allow MOG_IOSTAT_CMD env override (Eric Wong <normalperson@yhbt.net>)
+
+   * When a mogstored is down, die with a more informative message. (Dave Lambley <davel@state51.co.uk>)
+
 2012-06-21: Release version 2.64

    * Delete memcache data when we replicate fids (Pyry Hakulinen <pyry@automattic.com>)