A web crawler oriented to infosec. The original project was on sourceforge: http://sourceforge.net/projects/webcrawler-py.