Robots.txt is a text file residing in the root directory of your website that gives instructions to search engines as to which pages of your website they can crawl and add to their index.