monitor skips hosts marked dead or down

Host don't define readability/writability themselves, so the
Host::should_get_new_files sub is renamed to "alive" and
Device->can_read_from respects host status.

Also, queryworker now skips down/dead hosts in cmd_get_paths.

ref: http://code.google.com/p/mogilefs/issues/detail?id=46
Device->observed_* all respects observed host state

The observed unreachable state of the host should always supercede the
observed state of the device.  This is already the case with
observed_writeable, but not with observed_readable nor
observed_unreachable.

The monitor worker does not (and should not, to save bandwidth) update
states of all devices when a host goes down.
Device->should_read_from respects all host/device states

should_read_from() should replace all uses of can_read_from() in
non-Monitor workers.  This avoids the overhead of needlessly
rechecking devices either the monitor or user marked down.

This simplifies queryworker logic a bit.
DevFID size/checksum respects Device->should_read_from

Avoid needlessly attempting connections for checking files on
host/devices the monitor (or user) marked as unreadable.

This also makes the Fsck->size_on_disk function redundant.
get_paths: deprioritize devs in "drain" state

URLs pointing to devices set to drain are undesirable.
Files may disappear off draining devices immediately
after we've queried the file_on table and invalidate
the paths the client sends us.
t/00-startup: explicit fid test for create_open/close

This is mainly to prevent bugs like the fix in
commit ac5534a0c3d046e660fa7581c9173857f182bd81
from popping up again.
connection/mogstored: remove sock_if_connected()

This subroutine has been unused since MogileFS 2.52
commit 18a40d29236a8a00593f914191c9fd16cda082db
("Throw out old HTTPFile->size code and use LWP")
test for existing case-insensitive list_keys behavior

We cannot break existing case-insensitive behavior for list_keys
right now, even if it's broken.  This means SQLite/MySQL will
use case-insensitive LIKE statements for list_keys and Postgres
remains case-sensitive.
implement "case_sensitive_list_keys" server setting

Enabling this boolean will make the "after" and "prefix" params
of "list_keys" behave case-sensitively.

If this setting is /not/ enabled, clients will hit
after_mismatch errors when iterating through keys if they are
using an uppercase "prefix" argument and a subsequent list_keys
is called with an "after" that only matches case-insensitively.

If unset, this defaults to false (0) to match historical (buggy)
behavior.  Historical behavior is preserved (even if broken) as
users with small namespaces may rely on case-insensitive
matching.

Postgres users are not affected by this change, as the LIKE
operator in Postgres is always case-sensitive.

This change is tested on all three databases: Postgres, MySQL,
and SQLite.
t/02-host-device: unit tests for device/host state checks

Device state reporting functions should respect whatever the
underlying Host state is.
Delete memcache data when we replicate fids

Because memcache TTL is now user configurable, data in memcached might
be valid for a long time, and as such invalid paths might be returned.

It would be possible to populate memcache, instead of just removing. But
it might be wasteful when a device is marked as dead, those replicated
fids might not need to be in memcached.

There is still one TODO left. If someone modifies mindevcount and runs
FSCK, then the mappings might become incorrect, but I reasoned that it
would be rather rare.
Checking in changes prior to tagging of version 2.64.

Changelog diff is:

diff --git a/CHANGES b/CHANGES
index 07790f3..c552089 100644
--- a/CHANGES
+++ b/CHANGES
@@ -1,3 +1,15 @@
+2012-06-21: Release version 2.64
+
+   * Delete memcache data when we replicate fids (Pyry Hakulinen <pyry@automattic.com>)
+
+   * implement "case_sensitive_list_keys" server setting (Eric Wong <normalperson@yhbt.net>)
+
+   * get_paths: deprioritize devs in "drain" state (Eric Wong <normalperson@yhbt.net>)
+
+   * make marking a host down cause devices to act as down (Eric Wong <normalperson@yhbt.net>)
+
+   * monitor skips hosts marked dead or down (Eric Wong <normalperson@yhbt.net>)
+
 2012-05-29: Release version 2.63

    * Critical bugfix for Postgres users introduced by 2.61. New file uploads
When a mogstored is down, die with a more informative message.
iostat: allow MOG_IOSTAT_CMD env override

This makes it easier to test mock or alternative iostat
implementations.  This can be used for emulating the
iostat output on other platforms.
iostat: increase flexibility of iostat parser

The parser now looks for contiguous lines of statistics and (if
it's previously captured stats) emits whenever the first
non-stats line appears.  Relying on the "Device:" line is not
portable to FreeBSD (and possibly other iostats implementations).
The parser also ignores leading/trailing whitespace on each
statistics line.

Tested on Linux (sysstat 10.0.5) and FreeBSD 9.

For testing iostat output on FreeBSD, I used MOG_IOSTAT_CMD
like this on my GNU/Linux system:

  MOG_IOSTAT_CMD="ssh fbsd9vm iostat -dx 1 30" mogstored ...

ref: http://code.google.com/p/mogilefs/issues/detail?id=9
remove old rebalance knobs from server settings

This hasn't been used since the old rebalance code was nuked
for 2.40 (commit 0be2f9771a213e92d807f29bead13e6e4203ad54)
fix tests when /etc/mogilefs/mogstored.conf exists

mogstored gains a --skipconfig switch which we use in tests to
ignore the default config file.  mogilefsd has had this switch
(with identical semantics) since 2004.
tests: add basic test for reaper

Reaper isn't tested anywhere else.  We plan on changing it
slightly so ensure we don't introduce regressions.
reaper: factor out reap_fid sub from the work loop

A subroutine with a 20-line comment deserves to be its own sub.
This will make it easier to see what the future reaper lock
will guard without needing to scroll on small terminals.
reaper: global lock around DB interaction

We don't want multiple reaper process stepping on each other
during UPDATE/INSERT, causing needless conflicts/failures at the
DB level for every single FID.

JobMaster already locks its queues in a similar way to prevent
conflicts, so this should not noticeably harm performance (and
may improve performance due to the DB conflict reduction).