Improved the robots.txt download scripts
Update test examples with current robots.txt
Better handling of UTF8 BOM mark
Removed (hlint) warnings
Properly ignoring sitemap