add a bunch of documentation to HttpCrawlerClient, and implement throttling on a per client basis, derived from the crawl-delay in the domain's corresponding robots.txt file