What is a robot.txt file?
Robots.txt file is a file that contains rules for web crawlers. It specifies how to inform web robots (search engines) of what should be processed vs what should not be processed.