Fixed multiple disallow directives
Added several non-official directives, namely request-rate
Added bash script to update robots.txt files
Comments can appear alone in lines
Improved the robots.txt download scripts
Update test examples with current robots.txt
Better handling of UTF8 BOM mark
Removed (hlint) warnings
Properly ignoring sitemap
Merge pull request #6 from jcristovao/request-rate

Passing all tests