Fix multiple races between next_in_queue(), balanced_pool() and queue_empty().

This should fix some cases of 'stuck_in_queue' for items at the moment
they become eligible to build, and some "Unknown stuck queue bugs" when
only 1 or a few items remains to be built.

queue_empty(): Locking on balance_pool is pointless. Avoid races here by retrying.

  If a directory is found to be non-empty then we know the queue
  is not empty. Return immediately. If all dirs are found to be
  empty, check again to be sure nothing was moving around
  during the check. The only other writers are:
   1. clean.sh moving items from deps/ to unbalanced/
   2. balance_pool() moving items from unbalanced/ to N/

  This allows adding unbalanced/ back into the pool dirs. This fixes a
  race where balance_pool() may miss files added into unbalanced/ because of
  a child clean.sh moving files in after it had taken the lock and globbed
  the dir. The other child would then fail to grab the lock and not
  be able to run balance_pool(). The locked balance_pool() then would unlock and
  return. The queue would then be left with a lost item in unbalanced/
  that would not get processed. This lead to a "Unknown stuck queue" bug
  error.

next_in_queue(): Avoid races by retrying if failed to move an unbalanced item.

  Because next_in_queue() does not lock the pool dirs, it is possible it
  will find an item in unbalanced/ that another process is moving into
  an N dir via balance_pool(). If the race is lost, just try again and
  look in the N dir.

This effectively reverts [1b2b67db7e] and [2acab3a89f] and addresses the issue
differently.