Multiple improvements to the web crawler
Merge branch 'master' of https://github.com/sam-b/DataSearch

lalala