Robots.txt is a text file that instructs web crawlers about the pages on your site to crawl or not to crawl.