Multiple improvements to the web crawler
Multiple improvements to the web crawler