A multi-threaded, database assisted, web crawler.