store links to be crawled in a queue and use a loop to keep stack size constant
prefetch pages in multiple threads, but process them in the main thread. about 10x faster
clean up
remove unused file
marginally improve link handling
some minor domain scheme handling
add some error handling and update readme
update readme