added ability to resume where last crawl left off, which is useful if the crawling process is killed for some reason.
ignore logs