seomoz/irobot — pull request #1 by younker (2014-03-29 08:47:13+00:00)
Title: Irobot: initial review

As in Isaac Asimov's 'I, Robot'

This is a total re-write of https://github.com/fizx/robots which we currently use in OSE to request and parse robots.txt. The problem is it has not been touched for 3 years and we needed more control (config) and a way to cache results (a need stemming from the becu issue last week).

Initially I just forked it and started hacking on it but my ocd kicked in and the refactoring began. The one thing I have not refactored is the robots.txt parsing (txt_parser.rb). Now that I have specs in place to ensure there are no regressions, I can do that. However, I would like to get this ready for https. Probably not in time for launch, but shortly afterwards would be nice.

Any comments would be appreciated. This is by far the biggest single chunk of ruby development I have done in a long, long time.

#### TODO (not necessarily before merge)
- [ ] Clean up `txt_parser.rb`
- [ ] Fix the bugs I found in the existing parser (they have pending specs)