Basic working crawler!:
Bugs:
-no support for errors in retrieval (from broken links, forbidden links, etc.)
-does not currently distinguish between different depths
Fixed support for HTTPErrors
Note: Still encountered other parsing errors which crash the program
Insanely Slow Version
Plus side: no duplicate links!
Note: even with duplicate links (and getting the links from the same pages over and over
again), the previous version crawled faster