Hacked in support for crawl delay, removed some duplication in parser, added web mock … quick and dirty