web.py: Fixes clearing robots redirect url before it can be used.

Re: chfoo/wpull#58

When next_request was called multiple times, the redirect URL was set to
None. This caused it to fetch the same redirecting robots.txt repeatly.
changelog: Adds points about broken robots.txt and port number in hosts.