Robots.txt is a text file that helps webmasters to instruct robots how to index and crawl pages on their website.