Slightly modified web crawler from  http://code.activestate.com/recipes/576551-simple-web-crawler/
many tweaks
Added graphviz (dot) output format, excluded URL prefixes, URL "condensing" and cleanup to various filtering steps.

Graphviz output uses a huge hack.  To avoid syntax errors resulting
from using URLs as node names, URL nodes are identified by their md5
hashes (but still labelled with the actual URL.) The output looks
fine, but the graphviz file is very opaque.

"Condensing" rewrites URLs without fragment identifiers, so
http://foo/bar#baz and http://foo/bar#qux are understood to be the
same page.

Prefix exclusion is just a convenience for me: There's a bunch of
JavaDoc pages I want to ignore.
cleanup and comment code
Added README, initial Github version
Single line change to get it working with BeautifulSoup4
Merge pull request #1 from devinberg/patch-1

Single line change to get it working with BeautifulSoup4
Initial commit of AutoGen source code
Merge branch 'master' of github.com:cliffcolon/AutoGen_Webdriver_POs