temoto/robotstxt.go — pull request #7 by PuerkitoBio (2012-10-01 03:18:33+00:00)
Title: Fixes #6

This is kind of a big pull request, as documented in issue #6, it fixes how the library interprets groups of rules, according to [this paper of google's interpretation of robots.txt files](https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt).

I went ahead and changed parser and scanner to private, since there were many changes anyway in the implementation of the parser (scanner remained unchanged, except for the visibility). I also removed the error return value on Test[Agent] functions, since it was of little use as previously mentioned in my comment.

It adds support and features for:

* Google-compliant interpretation of robots.txt, with only the most specific group of rules applying to a given user-agent
* Google-compliant interpretation of rules, only the most specific path rule applies
* Support for patterns in allow/disallow directives
* Ignore empty directives, as recommended by the spec
* Support for sitemap directives
* Support for crawl-delay directive (non-standard extension, documented on [wikipedia's paper](http://en.wikipedia.org/wiki/Robots.txt)
* Whole suite of tests based on google's paper examples

I didn't squash my commits, so please feel free to get back to me if you'd like me to arrange things a little before merging. Same goes for any issue you may have with my code, I have no problem putting some more work into this to get it to fit with your style and vision. All tests are green.

Thanks!