Robots.txt is a text file that website publishers create and save at the root of their website. Its purpose is to tell automated web crawlers such as search engine bots which pages not to crawl on the website. This is also known as robots exclusion protocol.