"Robots.txt, also known as``Robot Exclusion Standard""or``robots.txt protocol"", refers to a standard text file that instruct search engine robots how to crawl and index webpages."
Through robot.txt you can instruct a search engine not to crawl and index certain files or directories within your site.