Some separation of concerns.

1) Crawler only makes requests and parses pages.
2) queueManager keeps a list of pages we need to visit as well as a list of pages we have visited,
   pages we failed to visit, and pages disallowed by robots.txt.
3) urlManipulator does things like getting the url for robots.txt, getting just the path from a url.