hook.py: Updates docstring for get_urls().
http.py: Adds url_info attribute to Response.
Adds http.RedirectTracker.
processor,robotstxt: Switches to using RedirectTracker.