During its visit to a website, a search engine web crawler (bot or spider) downloads the robots.txt file and extracts the links to find all pages linked to that domain. In order to determine which web pages should be listed in search engine results, search engine crawlers use rules and algorithms. They also determine how often the web pages should be crawled.