removed the crawlManualURLs function