tests: add test for fsck functionality

Before we make changes to the fsck code, we should ensure
we don't break existing use cases.

Behavior I'm uncertain about is documented with "XXX".
tests: fix fsck test to work on older LWP::UserAgent

The LWP::UserAgent module found in my Debian 6.0 installation
does not have a "delete" convenience wrapper.
add fsck test to MANIFEST
tests: fsck test use MogileFS::Store API when possible

No point in using DBI directly if a task can be done directly
via the MogileFS::Store API.

Noticed-by: dormando
t/60-fsck.t: fix overly long sleep when waiting for fsck log

This was leftover when I was monitoring the test with DEBUG=1 :x
t/60-fsck: additional test cases

* ensure fsck can handle a stall from an unresponsive mogstored

* ensure over-replicated files are cleaned up

* ensure fsck can work correctly with dead devices if it beats
  reaper to an FID
fsck: log bad count correctly instead of policy violation

A BCNT error is more descriptive than a generic POVI entry
and more accurately reflects the change made to an FID entry.

This also removes a dependency from /using/ the devcount column
and simplifies the code.  The devcount column remains invaluable
and informative to users, but MogileFS should not trust it for
making decisions when it has access to the file_on table.
additional tests for fsck stop, resume and stats

We need to ensure fsck can resume and returns sane
stats output.
t/60-fsck: retry SQL statements on deadlock

Hopefully this can eliminate some random test failures.
t/60-fsck: allow fsck_highest_fid_checked to be zero

After resetting, fsck_highest_fid_checked ends up at zero.
t/60-fsck: fix typo resulting in useless check

We checked the incorrect return value, so the second
mogstored failing would've gone unnoticed.
t/60-fsck: fix potential race conditions

These race conditions were causing this test fail occasionally.
These test failures were more common on SQLite and Postgres,
but not unheard of when using MySQL.

Some of these race conditions were due to fsck/job_master not
receiving settings changes in time, so we now resort to killing
existing processes and forcing them to reload.
fsck: update devcount, forget devs on unfixable FIDs

Whenever an FID is unfixable, be sure to update devcount (to
zero) to easily inform the user via mogstats.  If the FID
magically reappears in the future, the desperate fallback mode
will still find the file.
fsck: cleanup and reduce unnecessary devcount updates

fix_fid(): we no longer rely blindly update devcount on every
call.  This is important because we call fix_fid() on checksum
checks regardless, and devcount updates entail unnecessary
updates to the `file' table.

While we're at it, consolidate the places where we check the
skip_devcount flag and log bad devcounts.
t/60-fsck: detect fsck stalls via '!watch'

Instead of blindly sleeping, we can '!watch' through the tracker
and detect the the error message fsck sends.
t/60-fsck: use "!to <job> :shutdown" to kill workers

This is a simpler implementation and lets us be notified of
worker death (and pending replacement) as soon the tracker
notices it.
fsck: prevent running over 100% completion

FIDs may be created while fsck is running, causing "mogadm fsck
status" to report completion above 100% (and thus confusing
users).  Stopping fsck when it reaches fsck_fid_at_end (set to
MAX(fid) at fsck startup) prevents this.

This change should also have a pleasant side effect of reducing
contention with replicate workers on newly-uploaded FIDs.

ref: http://code.google.com/p/mogilefs/issues/detail?id=50
remove redundant code to ignore SIGPIPE

MogileFS::Server and Perlbal both ignore SIGPIPE for
us.  So there's no need to ever ignore it for socket
writes in HTTPFile, either.
delete: prevent orphan files from replicator race

Use the replicate lock here to prevent an DevFID from being
orphaned by a replicate process.  This prevents orphaned
requests if a user issues a delete request on a file while
replication worker is copying it.
don't attempt to broadcast undef config values to children

This avoids an uninitialized value warning from Perl when
choosing a value for the deprecated listen_jobs value.  Neither
the child nor the parent processes are capable of handling undef
values from :set_config_from_*.