scrapes the downloaded index pages and sets up next stage of the crawl