Web crawler needs comments + some technical fixes, multithreaded crawler only somewhat functional