Crawlers to have page and api urls