Clone of https://code.google.com/r/acrocrawler-crawler4j/ - attempt to improve various outstanding issues