fix "fsck never stops" issue

also probably an issue with rebalance. was losing the key for remove events on
server_settings
use cached setting for slave list

also switches to "confess" for DB connect failures. otherwise impossible to
tell which call tried to access a DB it shouldn't be talking to.
use DB server_settings if none are cached yet

allows startup code to call "server_settings_cached" without having to wait
for the monitor worker to run once.
Back off if master DB is down

If monitor worker figures master DB is down, send a flag telling workers to
avoid attempting to connect to the master. If a worker is already connected
and it functions, it will ignore the flag. If a worker is starting up or
reconnecting to the master, it will avoid the attempt.

After this commit a timed-out master DB will finally not cause the tracker to
fail, and reads will continue to work.

Takes almost a minute to notice the master is gone though. some room for
improvement there.
New variable that disables devcount updates

Devcount field is only useful for statistics and once you have enough
files it's not that useful even for that. Downing a device generates
less I/O load on the DB when devcount updates are disabled, there aren't
any queries against the file table.

Correct devcount can be restored by disabling the flag and running fsck.
Some storage servers can create full upload path during upload

New setting that allows admins to skip MKCOLs against storage nodes. Some
servers (nginx) can create can create full directory tree when a PUT request
is made. This will make cmd_create_open little bit faster because trackers do
not need to connect to storage nodes.

Needs this in nginx:
create_full_put_path on;
Filter out illegal memcache servers from memcache_servers

If you set incorrect memcache servers using mogadm:

mogadm settings set memcache_servers failure

Tracker query workers will start dying with:

Use of uninitialized value $ip in hash element at /usr/share/perl5/Cache/Memcached.pm line 253.
Use of uninitialized value in subroutine entry at /usr/share/perl5/Cache/Memcached.pm line 288.
Error: Bad arg length for Socket::pack_sockaddr_in, length is 0, should be 4 at /usr/lib/perl/5.10/Socket.pm line 214.

[Mon Jan 30 14:23:24 2012] [queryworker(29561)] Error running command 'get_paths': Bad arg length for Socket::pack_sockaddr_in, length is 0, should be 4 at /usr/lib/perl/5.10/Socket.pm line 214.
Unset memc when memcache servers have been removed

This allows one to get the client and see if memcache is being used or not e.g:

my $memc = MogileFS::Config->memcache_client;
if ($memc) {
  ..dostuff..
}
Remove FID from memcache when it is deleted

Removes domain-dkey -> FID mapping from memcache when a file is deleted.
Memcached LRU will take care of FID -> devid mapping.
Populate memcache even when noverify is missing

Because memcache is faster than DB, it makes sense to try to populate memcache
every time we fetch the values from database.

This also adds a configurable TTL for the cache, making it possible to cache
the values for much longer, thus making caching actually useful.
fix skip_devcount check from Pyry's commits
enable TCP keepalives for accepted sockets

Some clients may remain connected and idle for hours/days at a
time.  If their connections drop due to catastrophic failure and
the TCP implementation is unable to notify the server, the
server could eventually run out of file descriptors and/or
memory for new clients.

This change allows castastrophic network failures to be detected
after roughly 2 hours on most setups.  Sysadmins are free to
tweak the TCP keepalive knobs in sysctl or similar.
replicate: retry on short writes to destination

Neither write(2), sendto(2), nor any similar syscall is ever
guaranteed to have write-in-full behavior on sockets, so we need
to manually retry on unwritten portions.  This is difficult to
reproduce consistently (because the sockets are already in
blocking mode), but occasionally gets triggered on weak
networks.
note written bytes on replication PUT failure
reduce UPDATE's to device table

each tracker would update all device rows every 15 seconds. For setups with
lots of trackers, lots of devices, or both, this could end up with a huge
number of UPDATE's per second.

Now it tries pretty hard to update each row at most once every 15 seconds, at
the expensive of one extra DB read per monitor cycle.
Checking in changes prior to tagging of version 2.58.

Changelog diff is:

diff --git a/CHANGES b/CHANGES
index 46306bc..824e4a2 100644
--- a/CHANGES
+++ b/CHANGES
@@ -1,3 +1,29 @@
+2012-02-28: Release version 2.58
+
+   * reduce UPDATE's to device table (dormando <dormando@rydia.net>)
+
+   * replicate: retry on short writes to destination (Eric Wong <normalperson@yhbt.net>)
+
+   * enable TCP keepalives for accepted sockets (Eric Wong <normalperson@yhbt.net>)
+
+   * Populate memcache even when noverify is missing (Pyry Hakulinen <pyry@automattic.com>)
+
+   * Remove FID from memcache when it is deleted (Pyry Hakulinen <pyry@automattic.com>)
+
+   * Unset memc when memcache servers have been removed (Pyry Hakulinen <pyry@automattic.com>)
+
+   * Filter out illegal memcache servers from memcache_servers (Pyry Hakulinen <pyry@automattic.com>)
+
+   * Add skip_mkcol setting for servers which do not need it (Pyry Hakulinen <pyry@automattic.com>)
+
+   * skip_devcount variable that disables devcount updates (Pyry Hakulinen <pyry@automattic.com>)
+
+   * Back off if master DB is down (dormando <dormando@rydia.net>)
+
+   * use cached setting for slave list (dormando <dormando@rydia.net>)
+
+   * fix "fsck never stops" issue (dormando <dormando@rydia.net>)
+
 2012-01-31: Release version 2.57

    * Silence a warning about undefined value introduced by the slave database ordering work (hachi@kuiki.net)
Postgres wasn't honoring a no-wait timeout

passing a 0 timeout to get_lock means "try and give up immediately"
monitor shouldn't unlock unless it got a lock
don't make SQLite error out on lock calls

all of the proper wrappers return 1, so lets make the direct calls callable.
postgres: fix syntax error when getting lock

Introduced in commit 6c32b4c1f4e3ee83e85e049e179d226790d2683d