Added monitoring of seed offspring. Previous commit already added
tracking of the seed document ID for each retrieved page. This commit
adds a database to WorkQueues that couples each docid with a count
indicating the number of offspring in the queue for this page. As a
result, inProcessPages is now non-optional and is even used in
non-resumable crawls as this process requires knowing which pages are
assigned to crawlers at any time.

In addition, added a small patch to CrawlController that transfers the
crawl queue to a newly spawned thread in case of thread failure. The
reason is that these pages have been added to the inProcessPages queue
but will never be removed if a thread crashes because the list is then
dropped. This fix fixes this. Shouldn't happen either way, a crashed
thread indicates a bug of course. This change could potentially lead to
a repeatedly crashing thread if the cause is one of the retrieved pages,
however, this makes it easier to debug the issue itself.

A new method, handleSeedEnd(int docid) is added to WebCrawler. This may
be overridden by subclasses to do something when a seed has finished
crawling.