Fixed a bug where the crawler would crawl more than max_pages no of pages.