refactors robots.go and adds PoliteGet() method
refactors Page, adding comments and clarifying function names
adds worker
adds StagedCrawl
adds TestGetPage() for Worker.GetPage()
adds TestGetPages() for Worker.GetPages()
adds unit tests for StagedCrawl()
adds InfiniteCrawl() which uses Worker and replaces Crawler