A robots.txt file is a text file that provides instructions to robots (search engine crawlers) regarding the pages on your site that should be crawling and those they should not.