Correctly include the link for directory1/doNotCrawl.html
Allow for wildcards in a disallow path

rules like `Disallow: /*/book/export/html/` were not working because the wildcard was being encoded.

This fixes a regression, as this used to work.
Merge remote-tracking branch 'upstream/master' into robots
Merge pull request #31 from mfairchild365/robots

Fix regression