Fix when the download status code can not be determined
Fixed a bug when the meta tag robots does not contain 'noindex','index' or 'all'
Improved the robots.txt implementation for malformed robots.txt-files. Also the system tries to find a sitemap.xml if no one is mentioned in the robots.txt file