Refactor the web crawler a bit