Robots: Catch exceptions that the downloader might throw

Consider any exception to mean that the request failed.  The prevents the scan from failing just because we couldn't get the robots.txt file.