crawl delay support

hacked in support for crawl-delay
changed tests to use webmock and a local copy of the robots file
removed some duplication in the parser

- this was quick and dirty, but it is tested (if you can call 1 test, tested)