reduce rebalance delays

This doesn't fix bug #724, but makes rebalance more usable.
don't warn on SQLITE_LOCKED in DB checkpoint
test/start-nginx.sh: support setting up a cluster with N nodes
GC implementation

Squashed commit of the following:

commit cc7cf312fb79721335b98e2d70e91d7a8e63702c
Merge: a3e8de5 ff85a3e
Author: Török Edwin <edwin@skylable.com>
Date:   Thu Aug 21 14:32:52 2014 +0300

    Merge remote-tracking branch 'origin/gc'

    May conflicts, beware that commit 2fb8fa2d9760656012bbf8b04e6a33d094f52c4f from master
    contains some unspecified cherry-picked and squashed commits from the origin/gc branch,
    at least these: 486c88f02e80c72b9cbbae7b3d2a597af8cec047 72947d7e36bd24603957e5f0828e04113aa3b04f

commit ff85a3e909b32021befa517f77a071b04a0f5b46
Author: Török Edwin <edwin@skylable.com>
Date:   Mon Jul 28 12:56:05 2014 +0300

    keep negative counts too (needed during rebalance)

commit 45646408ad8369241fc8896103e530a1544d365a
Author: Tomasz Kojm <tkojm@skylable.com>
Date:   Tue Jul 15 12:37:08 2014 +0200

    fix help page

commit 675e1fd4f5d2f60d76b1dd5c096a7901aad0e73b
Author: Török Edwin <edwin@skylable.com>
Date:   Mon Jul 14 16:41:09 2014 +0300

    fix an uninit in a DEBUG() message too

commit eb38d7cd5686e6ae8b28fe5e302664de0a9f9b66
Author: Török Edwin <edwin@skylable.com>
Date:   Mon Jul 14 16:34:32 2014 +0300

    fix uninit error reported by valgrind (bb #665)

commit 486c88f02e80c72b9cbbae7b3d2a597af8cec047
Author: Török Edwin <edwin@skylable.com>
Date:   Wed Jul 9 10:42:53 2014 +0300

    GC reservation handling (bb #651)

    note: this introduces a race condition between sxrm and sxcp by design.
    Start 2 sxcp for same file(name), and after the first finishes then sxrm it.
    Now if the GC runs before the 2nd sxcp finishes then it will fail because the
    hashes are no longer reserved and they got removed.

commit e525edb59b65ab173014ef430ff699d54aef92d0
Author: Török Edwin <edwin@skylable.com>
Date:   Thu Jul 3 23:21:24 2014 +0300

    GC tokens and operations separately

commit 04a2c1dc81bdbed17bcf1ae2a8a3ce3d18b8b321
Author: Török Edwin <edwin@skylable.com>
Date:   Thu Jul 3 15:21:27 2014 +0300

    GC: don't wait for reservation to expire if hash was marked as inuse/deleted
    already

commit feee5d96e4134fa2fc4fc6f6b3f65dabe662f596
Author: Török Edwin <edwin@skylable.com>
Date:   Thu Jul 3 14:09:50 2014 +0300

    GC: avoid N^2 iteration

    resume iteration from where we left off, not the beginning.

commit c4c6e969c6354f7ddb6c0a1c170e405529edc3de
Author: Török Edwin <edwin@skylable.com>
Date:   Thu Jul 3 12:21:35 2014 +0300

    GC hashes that were never used and their reservation expired too

commit a5166325d5648537a1fdfc3085b35c9a1d10a3b1
Author: Török Edwin <edwin@skylable.com>
Date:   Wed Jul 2 18:51:41 2014 +0300

    fix leak in parsing inuse queries

commit 62b723e3c84919601d62dbfc2f34933aa05be50a
Author: Török Edwin <edwin@skylable.com>
Date:   Wed Jul 2 16:04:31 2014 +0300

    print GC statistics

commit a6b0ffd9b5a29e9be1446e64734a8345472cc6c0
Author: Török Edwin <edwin@skylable.com>
Date:   Wed Jul 2 15:17:00 2014 +0300

    make GC interruptible

commit a67b9be3108d7320a46ecd1b697cfda1a825111e
Author: Török Edwin <edwin@skylable.com>
Date:   Tue Jul 8 11:59:20 2014 +0300

    GC: per-operation expiration time

    Conflicts:
            server/src/fcgi/blockmgr.c

commit fd6cf69358bec23955b8d17d7649e2497b90378b
Author: Török Edwin <edwin@skylable.com>
Date:   Tue Jul 8 11:58:44 2014 +0300

    GC iteration/checkpoint improvements, set blocks as free

    Conflicts:
            server/src/common/hashfs.c

commit 72947d7e36bd24603957e5f0828e04113aa3b04f
Author: Török Edwin <edwin@skylable.com>
Date:   Tue Jul 1 17:18:14 2014 +0300

    increase file delete job timeout

    similar to file flush job timeout, based on no of blocks and nodes

commit 49b67858810d04192963dff53bdc6b78ce4a6a9c
Author: Török Edwin <edwin@skylable.com>
Date:   Fri Jun 27 17:21:28 2014 +0300

    Drop unused functions from old GC

commit 3c8e02af4b05593a8e9f11484f25a3f7e2f006f4
Author: Török Edwin <edwin@skylable.com>
Date:   Tue Jul 8 11:57:59 2014 +0300

    New GC WiP

    Conflicts:
            server/src/common/hashfs.c

commit df40ad1a42c810759a4f7276d661908558d02849
Author: Török Edwin <edwin@skylable.com>
Date:   Fri Aug 22 12:06:48 2014 +0300

    use can_gc

    fix can_gc: it was always returning OK
commit 0d4993437e67983f46f9a41f27be645def6f45b2
Author: Török Edwin <edwin@skylable.com>
Date:   Fri Aug 22 12:57:58 2014 +0300

    debugging

    debugging: turn off deletes of reservations/tokens
    more debug code

    re-enable reset of GC

    restore the 'rebalance in progress' sleep to 60

    if sxsetup is used properly (with --wait) this should never be reached.

    fixup forced awake sleep time

commit 06cc6157674df286506aa6f40e87e85372600cd3
Author: Török Edwin <edwin@skylable.com>
Date:   Tue Sep 2 16:04:46 2014 +0300

    fundamental GC fixes

    fix hash presence check for rebalance counter API

    avoid race condition on file flush between reserve and inuse

    When a hash is marked as inuse we must delete its reservations.
    However we must only delete the reservation for the current hash, not for
    all hashes of the current file, otherwise there is a race condition:

    file upload N hashes
    replicated M<N hashes
    file flush
    try to mark as inuse all N hashes, deleting all reservations
    GC runs : deletes the placeholder for the (N-M) hashes that were not yet
    replicated
    blockmgr tries to replicate missing (N-M) hashes, but it can't because
    reservations are gone

    GC files deleted after rebalance properly

    We either have an entry in the blocks table, and no entries in the use table,
    OR we have entries in the use table that sum up to 0.
    For example if a block is NOT moved during rebalance and then deleted after
    rebalance we'd get: age=2, use=1; age=3, use=-1 => sum(use)=0.

commit e78ac5a5e4b297a400b25622875fef52664587c1
Author: Török Edwin <edwin@skylable.com>
Date:   Wed Sep 3 13:01:28 2014 +0300

    ENOENT related hash counter fixes

    detect block upload failures due to missing reservation

    do not replicate blocks with inuse count = 0

    the remote node would reject the replication of such counters,
    and they would get deleted on the next GC run anyway.

    rebalance block counters: accept all counters

    Don't exit the loop on ENOENT when processing received hash counters,
    because during a rebalance the hash will not be there yet, but we do want to store all
    the counters.

    try to allow pushes of hashes that were not reserved

    TODO: check

    record rm -1 even if hash is not present

commit 38a3f53ecbb64324a91ddccf88687034af3e0cc9
Author: Török Edwin <edwin@skylable.com>
Date:   Tue Sep 9 18:37:41 2014 +0300

    fix reservation expiration by ttl
fix reservation expiration by ttl

commit cdd0e67338ef1ce086dbf2644a74ff0121bf34f1
Author: Török Edwin <edwin@skylable.com>
Date:   Thu Sep 11 11:40:38 2014 +0300

    don't hash the inuse query id again

    it must match the reservation id exactly

    reservation ID should be same regardless of volume node
commit 6e34f09721be94fa2b8c4ff349cd8c459d353c5e
Author: Török Edwin <edwin@skylable.com>
Date:   Thu Sep 11 14:55:04 2014 +0300

    test/start-nginx.sh: accept N= as env var, and use --wait
    test/start-nginx.sh: add a download test after rebalance
commit 7e5e11c5a4c9695dc193019b30572074d7077e47
Author: Török Edwin <edwin@skylable.com>
Date:   Tue Sep 16 16:48:49 2014 +0300

    make -X expire all reservations

    Even those without any blocks uploaded.
    Normally we wouldn't expire these reservations before their ttl, because we'd
    break uploads, but -X is meant to remove all reservations (and break uploads).

    do not expire operations with -X

    It is not safe, we might have a pending delete when -X is run
    that retries a block delete (decuse) after -X and we'd incorrectly
    allow the duplicate operation.

commit 83caed2436f0996a38fb26dceed3ab38e8091706
Author: Török Edwin <edwin@skylable.com>
Date:   Thu Sep 18 13:27:05 2014 +0300

    check gc_expire_trigger too

commit 5576c86ac95dc2a6dbcc88ce99e0e43ef508ff4d
Author: Török Edwin <edwin@skylable.com>
Date:   Tue Sep 23 10:39:00 2014 +0300

    restore GC_GRACE_PERIOD
add sxadm cluster -X support
add missing sxc_cluster_free()
Do not restore global error while getting error code/message from cbdata

Cosmetics. Put correct error message in debug entry.
decrement hash refcounts from all distributions

when deleting files during rebalance delete the blocks on the new set

Scenario:
file A on present old volnode set, not yet on the new volnode set
blocks of file A present on old set, and counters already pushed to the new set
counters(old set): 1, counters(new set): 1
sxrm is run, query sent to old volnode set
delete-by-rev queries are sent to both old and new volnodes from the old volnode
  delete-by-rev received by new volnode: file is not present -> delete-by-rev is a no-op here
  delete-by-rev received by old volnode: file is present ->
   check whether we are part of the new volnode set -> no
   check whether we are part of the old volnode set -> yes
     send deletes to hash blocks to the 'dist', i.e. old set
     counters(old set): 0, counters (new set) :1 -- wrong

after this patch
     * send deletes to hash blocks to the new set:
     counters(old set): .. as before, counters(new set): 0 -- correct

Note: the previous code worked correctly in the scenario where
file A's blocks are NOT pushed to the new set yet, and thus deleting
them on the old set prevents them from being replicated in the first place.

Situation is more complicated when the same node is a volnode both on old and
new sets, or if the file is already present on both old and new volnode sets.
In that case we send duplicate delete queries, and rely on the same mechanism
that we rely on during retries on file flush:
 each (replica,fileid,operation) tuple is only applied once, duplicate queries
 are a no-op

P.S. using NL_NEXTPREV or NL_PREVNEXT would be correct too