saltybeagle/Spider — pull request #30 by mfairchild365 (2014-04-01 16:37:22+00:00)
Title: Robots: Catch exceptions that the downloader might throw

Consider any exception to mean that the request failed. The prevents the scan from failing just because we couldn't get the robots.txt file.