Improved shutdown performance. Due to the now mandatory inProcessPages
thanks to the seed tracking, it is known whether any crawler is
processing a page or not. If inProcessPages is non-empty but any crawler
is not working, this is a bug that should be fixed. If WorkQueues is
non-empty but any crawler is not working, this is also a bug. Therefore,
it's safe to skip waiting. In addition, the timeout that waits before
cleaning up is now base on a join of the crawler threads instead of just
waiting 10 seconds. This is usually much faster and more safe.