added CrawlStateManager interface
implemented the crawl state management feature
Merge branch 'feature-track-runs-naive' into merge-feature-track-runs
removed unused libs
removed unused libs
added getExpiredUriIterator to crawlStateManager
added shutdown method to sink interface
added a check to avoid work after shutdown