Fix "skip_devcount" during rebalance

We were updating devcount field even when skip_devcount was true. We
should not use $sto here because we already have FID object and nice
method available for this.

Signed-off-by: Eric Wong <normalperson@yhbt.net>
fix use_dest_devs for rebalance

The caller expects an array ref, currently using use_dest_devs will kill
JobMaster with:

Oct 18 09:45:25 storage22 mogilefsd[23263]: crash log: rebalance cannot find suitable destination devices at /usr/local/share/perl/5.10.1/MogileFS/Worker/JobMaster.pm line 233
Oct 18 09:45:26 storage22 mogilefsd[22044]: Child 23263 (job_master) died: 256 (UNEXPECTED)

Signed-off-by: Eric Wong <normalperson@yhbt.net>
prevent reqid mismatches (and queryworker death)

On certain errors, the queryworker may send two "ERR" responses, causing
the ProcManager to terminate the queryworker upon reading the second
response if the queryworker is immediately fed another query.

This can affect busy setups, but is also easy to reproduce with a single
queryworker that's receiving a pipelined request to an
invalid/non-existent domain:

	(
		printf 'list_keys domain=\r\nlist_keys domain=\r\n'
		sleep 2
	) | socat - TCP:127.0.0.1:7001

The queryworker strace will look like this (writing 4 lines):

  write(14, "4981-1 0.0005 ERR no_domain No+domain+provided\r\n", 48) = 48
  write(14, "4981-1 ERR domain_not_found Domain+not+found\r\n", 46) = 46
  write(14, "4981-2 0.0005 ERR no_domain No+domain+provided\r\n", 48) = 48
  write(14, "4981-2 ERR domain_not_found Domain+not+found\r\n", 46) = 46

And a message like this will appear for "!watch" users:

  Worker responded with id <undef> (line: [4981-1 ERR domain_not_found Domain+not+found]), but expected id 4981-2, killing

This is because ProgManager immediately calls NoteIdleQueryWorker upon
writing the first ERR response to the client (at the end of
HandleQueryWorkerResponse).  This means the idle query worker may
immediately start processing a second request before the ProcManager has
a chance to process the second ERR response line (from the first
request).

Preventing err_line() from calling send_to_parent() with "ERR"
if querystarttime is undef prevents this issue, but there may be
better ways to fix this bug.  A similar, preventative fix may be
appropriate for ok_line().
test: expose try_for() as a common test function

This saves us from reinventing it in every test and will
help us detect stuck tests more easily.
t/50-checksum.t: use common try_for() function

I've had this test get stuck intermittently in different
places, this should make it easier to track down stuck
tests.
t/50-checksum.t: ensure replicate worker is really down

Workers do not receive nor respond to messages as soon as
the ProcManager dispatches the request to stop/start them,
so wait until ProcManager no longer knows about a process
before proceeding.
checksum: avoid comparison on uninitialized value

$class->{hashtype} is undef by default for classes where no
checksums are configured.

Since clients can force checksum verification in create_close
regardless of class, we can end up with a Checksum object for
FIDs regardless of which class the FID is in.
query: allow "0" key on all commands which take keys

delete, file_info, get_paths, rename, file_debug, updateclass
were all broken when handling a key named "0".
fsck: use replicate lock when fixing FID

We need to ensure neither replicate (nor delete) are changing
the devids list when fixing an FID.  This should ensure we're
safely modifying the devid list for a given FID when forgetting
about bad ones.
fsck: skip non-existent FIDs properly

We should not waste time stat()-ing FIDs that no longer
exist at all.
improve handling of classes which change hash algorithm

In replicate, we validate via existing FID checksum regardless
of class.  This failed when the class.hashtype was altered after
uploading but before replication.

The following sequence of events caused replication to fail:

  1. class.hashtype = MD5
  2. FID created and stores MD5:...
  3. FID enqueued for replication
  4. class.hashtype = NONE
  5. FID begins replicating

Replication (Step 5) failed since the existing MD5 digest is
trying to validate against a (now) non-existent class digest.
Since we stored the checksum in the database anyways, calculate
and validate anyways as an admin could've only wanted to
alter a class temporarily, not permanently.

An admin may also decide to switch checksum algorithms.
Fsck now logs hash algorithm mismatches as "BALG" and
emits a descriptive message to syslog
reaper: validate DB connection before reaping

This helps prevent the reaper process from dying if
the DB disconnected us for idleness.

This should fix #75: ("reaper dies if DB connection closes")
http://code.google.com/p/mogilefs/issues/detail?id=75
t/30-rebalance: remove redundant try_for() function

I missed this when moving try_for() into Test.pm
t/00-startup: fix updateclass test

We called MogileFS::Client::update_class incorrectly without
the key.  Additionally, the test for checking the number of
copies was also incorrect.
support updating the class to the default class which has an id of 0

[ew: added trivial test]
Signed-off-by: Eric Wong <normalperson@yhbt.net>
add a hook to cmd_updateclass

Signed-off-by: Eric Wong <normalperson@yhbt.net>