Crawler overhaul in progress. Added precise tracking of CrawlInfo nd end of crawl status.