removed unnecessary onErrorCrawlingPage function
removed NUM_CRAWLERS config option, added crawlQueue check to shouldPageBeCrawled function