Catch exceptions that the downloader might throw

Consider any exception to mean that the request failed.
Merge pull request #30 from mfairchild365/robots

Robots: Catch exceptions that the downloader might throw