meanpath/robots — pull request #5 by jcristovao (2014-04-01 12:47:49+00:00)
Title: Fixed multiple disallow directives

Hi,
Congratulations for the quite useful package. However, I did found some bugs :p

It does not handle multiple disallow directives, like:
```
User-agent: *
Disallow: /db.php
Disallow: /submit.php
```
So, I started by making the examples test more strict: now they only pass if everything is parsed, not only if a Right value is returned.

Then I fixed the multiple disallow directives bug, included (ignored) the host directive.
Still missing is the handling of the request-rate proposed directive, that seems to be already in use in some places:
http://www.conman.org/people/spc/robots2.html#format.directives.request-rate
(currently 2/3 tests fail because of this).

Thanks
Thanks