You can crawl all HTML pages as well as websites that rely on JavaScript. Additionally, you can choose to crawl the website’s sitemap, ignore robots.txt rules, and follow internal NoFollow links.