Removed "CrawlEngine" and made it more like the old WebCrawler class.