Added a few more functions to the web crawler