Refactored so that one HttpClient is used for each crawl session.