fid size workaround for lighttpd 1.4.x

Fixes Issue 39. Lighttpd doesn't return a Content-Length header for zero byte
files. So we add a specific workaround for it. If it's not lighttpd, we should
still throw errors if that header is missing.
sqlite: fix mogdbsetup failure on new setups

upgrade_modify_device_size() should not fail on new setups.
SQLite won't enforce MEDIUMINT vs INT anyways, so it's safe
to always continue and store larger INT values in MEDIUMINT
columns.
Fix mb_free() counting empty device as missing
the exist method no longer exists

Update the if condition to match the if condition of the similar code blocks in the Delete Worker
Allow default class to be overridden again.

Also makes its support slightly more DWIM. If you ask to modify the default
class, it will be autoadded to the database if it hasn't yet. Add works as it
should.

Delete *doesn't work*. Once you add a default class to the database, you're
stuck with it. You can continue to change it. We could "delete" it and walk it
back to an original form, but it seems more correct to tell people to not do
that.
Issue 36: Further insulated from autoinc resets

If MySQL is restarted and tempfile loses its autoinc counter, normally
mogilefs will detect that a new fid insertion is a duplicate and resets it to
MAX(fid) FROM file. However this doesn't work as well when your low fids may
be deleted.

So now, trackers check MAX(fid) on start and refuse to go below that. Needs
further work but this will go a long way.
Checking in changes prior to tagging of version 2.54.

Changelog diff is:

diff --git a/CHANGES b/CHANGES
index 5e4e8d2..79e8a7c 100644
--- a/CHANGES
+++ b/CHANGES
@@ -1,3 +1,21 @@
+2011-10-29: Release version 2.54
+
+   * Issue 36: Further insulate from mysql restarts resetting the fid counter (dormando <dormando@rydia.net>)
+
+   * Allow default class to be overridden again. (dormando <dormando@rydia.net>)
+
+   * Fix crash in old delete worker code (Daniel Frett <daniel.frett@ccci.org>)
+
+   * Fix mb_free() counting empty device as missing (git.user <git.user@gmail.com>)
+
+   * sqlite: fix mogdbsetup failure on new setups (Eric Wong <normalperson@yhbt.net>)
+
+   * fid size workaround for lighttpd 1.4.x (dormando <dormando@rydia.net>)
+
+   * Fix various issues in the new mogilefs server query worker. (Tomas Doran <bobtfish@bobtfish.net>)
+
+   * Allow chunked PUT's of unlimited size (dormando <dormando@rydia.net>)
+
    * Fix listing FIDs.
    * Fix fsck fixing a file from logging a message which appears to be
      two commands (due to spurious \n). Also fix debug function to
Wrap the queue tx with a mutex lock under mysql

MySQL versions are very deadlocky with this pattern, unfortunately. Worse yet,
under 5.1+ INSERT's can block forever so long as the SELECT's overlap.

Wrapping it in a lock to serialize the transaction (which is fine for
performance, anyway) stops the INSERT's from blocking and reduces the
deadlocks a little.
restore new file upload replication priority

now that the queues are locked uniquely, and MySQL 5.1+ does better at
avoiding deadlocks, return to scheduling new file uploads as "0" (immediate)
so reaper work doesn't gum them up.
stop enforcing a source id on new file upload

file_on rows have been added ahead of the replication call since... forever?
so just use it. Over-replication bugs on new upload get jammed up due to this.

I just can't think of a justification for keeping it the way it was.
schedule fsck replications for the future.
return 1 from condthrow if all is well

many functions just return from condthrow, but we often check their return
values :/

Thanks to Eric Wong for the tip.
More stats to help troubleshoot rebalance

Need to be able to see more easily if a rebalance is jammed. Also fix a random
warning.
fixed sources in replication are now a suggestion

Code used to hard error if a fixed source was missing. That would be useful
if, say, FSCK decides that some copies are bad and it should be re-sourced
from a particular device. However FSCK can just nuke or mask bad rows and
re-run.

So... attempt, then give up. We can always revert the behavior if some feature
requires it. New file uploads no longer use fixed sources, which gets rid of
most of the issue. This change should let existing clusters clean themselves
up.
Checking in changes prior to tagging of version 2.55.

Changelog diff is:

diff --git a/CHANGES b/CHANGES
index 79e8a7c..d03029f 100644
--- a/CHANGES
+++ b/CHANGES
@@ -1,3 +1,19 @@
+2011-11-14: Release version 2.55
+
+   * fixed sources in replication are now a suggestion (dormando <dormando@rydia.net>)
+
+   * More stats to help troubleshoot rebalance (dormando <dormando@rydia.net>)
+
+   * return 1 from condthrow if all is well (dormando <dormando@rydia.net>)
+
+   * schedule fsck replications for the future. (dormando <dormando@rydia.net>)
+
+   * stop enforcing a source id on new file upload (dormando <dormando@rydia.net>)
+
+   * restore new file upload replication priority (dormando <dormando@rydia.net>)
+
+   * Prevent queue related MySQL query hangs (dormando <dormando@rydia.net>)
+
 2011-10-29: Release version 2.54

    * Issue 36: Further insulate from mysql restarts resetting the fid counter (dormando <dormando@rydia.net>)
worker: use timeout with read_from_parent() instead of sleep

Since read_from_parent() got an optional timeout to use with
select(), we don't have to blindly sleep and use a non-blocking
read anymore.
queryworker: reimplement "clear_caches" command

Technically, it "refreshes" the cache and usage information,
but the effect is the same with the new monitor.

By swapping do_monitor_round for clear_caches in the tests,
tests run about 30s faster on my machine.  Additionally,
all changes to devices/hosts/domains/classes should refresh
the cache on a given worker.
t/00-startup.t: remove 5-second sleep

It appears to no longer be needed.  However, due to potential
race conditions and the presence of multiple query workers, we
still need to clear_caches in some places.
don't clear state caches for cmd_update_class

command updates the class attached to a FID, not necessary to clear metadata
caches.
make reaper wait less time in test mode

actually makes anything not wait more time in test mode. since in all
automatic test cases we cause clear_cache to run before examining anything.