meanpath/robots — pull request #6 by jcristovao (2014-06-17 10:37:12+00:00)
Title: Passing all tests

Hi again,
In order to fully address #5 , some major work was needed. Namely:

* Now parses non-standard crawl-delay and request rate directives, merging them into a single directive
* Now handles UTF8 BOM (dropping it)
* Handles NoArchive, NoSnippet, NoTranslate, NoIndex
* Some fixes to newlines
* Various code simplifications (mostly following hlint suggestions).
* Bash script to update robots.txt examples
* Changed attoparsec imports to avoid deprecation warnings

The tests now all pass, except for nbcnews.com, which I believe is an improper robots.txt (to many *). I've moved it to bad-examples folder. I have also included updated versions of all robots.txt files in the ```tests/examples``` folders.

But, although all tests are passing, some changes are still necessary:

* `canAccess` does not check for time (request-rate / visit-time directives)
* SiteMaps not parsed - thinking about turning ```Robot``` into a full blown datatype to address this, and thus
* Major version number should be bumped.

I'm planning to address canAccess first, but if you agree these can be merged already.

Thanks,
Cheers,
João