A basic inspection type crawler designed to be used as part of website audit and investigation. Supports robots, crawl rate limits crawled pages limit, mime type limits and domain restrictions.