Robots.txt file is a file that contains rules for web crawlers. It specifies how to inform web robots (search engines) of what should be processed vs what should not be processed.