Don't send extra HEAD requests

The spider was sending extra HEAD requests to get the effective URL of links on a page.  That however was sending a ton of extra requests and adding a lot of load to remote servers.  Instead get the effective URL while downloading the page, and delete the page if we need to.
Return early because $this was deleted
Use a custom useragent
Merge pull request #58 from mfairchild365/redirects

Reduce load on remote servers