Merge pull request #30 from mfairchild365/robots

Robots: Catch exceptions that the downloader might throw
Correctly include the link for directory1/doNotCrawl.html
Allow for wildcards in a disallow path

rules like `Disallow: /*/book/export/html/` were not working because the wildcard was being encoded.

This fixes a regression, as this used to work.
Merge remote-tracking branch 'upstream/master' into robots