rinzi/robotstxt — pull request #4 by johnjansen (2013-03-04 19:57:23+00:00)
Title: crawl delay support

hacked in support for crawl-delay
changed tests to use webmock and a local copy of the robots file
removed some duplication in the parser

- this was quick and dirty, but it is tested (if you can call 1 test, tested)