Make WebCrawler more robust.
Merge branch 'master' of https://github.com/ChiTheHotDogGuy11/Distributed-Systems