Initial import of old TestLinks code - from 2007\!
Minimal changes to replace the deprecated htmllib.HTMLParser with newer HTMLParser.HTMLParser.
Use requests library to do a HEAD for http links, rather than urlopen and then close \(which looks to the server like an aborted GET\).
Better handling of relative links when the seed location is a local file.
Small general tidy up.
Be more explicit about failures by returning a message.\nAttempt to detect bad hostnames.
Handle https.
Handle URLs with missing schemas.
Abide by robots.txt
Handle https.
Handle fragments in URLs, e.g. anchors.
Default behaviour is now *not* verbose.  Added strict mode.
Don't choke on FTP / Gopher links.
Basic throttling: don't hit the same server more than once per 5 seconds.
Attempt to figure out when somebody forgot to put http:// infront of a target URL and do it for them.
Better handling of relative links.
Attempt to guess the root directory of targets on the local filesystem.